Drift-Resilient TabPFN: In-Context Learning Temporal Distribution Shifts on Tabular Data

Authors: kai  helli
david  schnurr
noah  hollmann
samuel  müller
frank  hutter
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 20:05:18

Abstract: While most ML models expect independent and identically distributed data, this assumption is often violated in real-world scenarios due to distribution shifts, resulting in the degradation of machine learning model performance. Until now, no tabular method has consistently outperformed classical supervised learning, which ignores these shifts. To address temporal distribution shifts, we present Drift-Resilient TabPFN, a fresh approach based on In-Context Learning with a Prior-Data Fitted Network that learns the learning algorithm itself: it accepts the entire training dataset as input and makes predictions on the test set in a single forward pass. Specifically, it learns to approximate Bayesian inference on synthetic datasets drawn from a prior that specifies the model's inductive bias. This prior is based on structural causal models (SCM), which gradually shift over time. To model shifts of these causal models, we use a secondary SCM, that specifies changes in the primary model parameters. The resulting Drift-Resilient TabPFN can be applied to unseen data, runs in seconds on small to moderately sized datasets and needs no hyperparameter tuning. Comprehensive evaluations across 18 synthetic and real-world datasets demonstrate large performance improvements over a wide range of baselines, such as XGB, CatBoost, TabPFN, and applicable methods featured in the Wild-Time benchmark. Compared to the strongest baselines, it improves accuracy from 0.688 to 0.744 and ROC AUC from 0.786 to 0.832 while maintaining stronger calibration. This approach could serve as significant groundwork for further research on out-of-distribution prediction.

Main paper: 
 introduction: 
 In traditional machine learning the train and test data are assumed to be sampled from the same distribution [1] . However, this assumption of independent and identically distributed (i.i.d.) data is commonly violated in real-world scenarios due to distribution shifts, resulting in performance degradation of standard machine learning (ML) models over time [1, 2] . Research in the area of temporal domain generalization (Temporal DG) tries to address these shifts by developing methods that perform consistently across temporal domains and generalize beyond the training regimen, i.e. into the future. In fields such as healthcare, climate science, or finance, data is most often organized in a tabular format [3, 4] . Here shifts are driven by hidden variables such as policy or climate changes, equipment updates, seasonal changes, or activity cycles, limiting real-world model deployment [2] .Young and Steele [5] describe declining mortality quantification in a hospital system while Pasterkamp et al. [6] show deterioration of cardiovascular risk models over time, leading to increased mortality; Ganesan et al. [7] show the COVID-19 pandemic exacerbated this issue, as ICU mortality prediction models failed to adapt to the unique characteristics of COVID-19 patients; environmental models need to continually adapt to climate changes [8] ; fraud detection models need to continuously adapt HIGH-LEVEL OVERVIEW (a) We generate synthetic datasets by sampling structural causal models whose edges shift over time.A structural causal model (SCM) with sparse edge shifts indicated in red.A 2 nd -order SCM generates the strength of edge shifts depending on a time domain. Given millions of datasets from (a), it learns to make predictions on held-out test samples, learning the prediction algorithm itself with an inductive bias for addressing distribution shifts.(c) Trained once, Drift-Resilient TabPFN can be applied to novel real-world data and makes predictions in a single forward pass, automatically detecting and extrapolating distribution shifts. Figure 1 : High-level overview of our method. We train a transformer that accepts entire datasets as input to learn the learning algorithm itself by training on millions of synthetic datasets once as part of algorithm development. The trained model can be applied to arbitrary real-world datasets. In (b), X, c, and y refer to features, time domain, and label respectively. In (c), we show predictions on test domains 4 (left) and 5 (right), where we see a distribution shift. Drift-Resilient TabPFN accurately updates decision boundaries in this example.as the strategies of fraudsters adapt to the models themselves [9] ; these feedback loops often arise in practice, where model deployment inherently causes a system change [10] .Robustness to such distribution shifts stands as a prominent challenge in current ML research [11] . So far multiple approaches have been proposed to address temporal distribution shifts using neural networks (NNs) [11] [12] [13] . However, modeling distribution shifts in tabular data presents a two-fold challenge: (i) NNs have struggled to model and extrapolate distribution shifts to date [11, 14] (ii) approaches for modeling distribution shifts have mostly employed NNs, while tree-based methods have consistently outperformed NNs in handling tabular data [3, [15] [16] [17] -leaving a wide methodological gap in addressing this common real-world scenario.We provide a fresh perspective on predicting given distribution shifts by leveraging in-context-learning (ICL) to learn the prediction algorithm itself -bypassing many challenges encountered in this setting. Our approach builds on the foundation of Prior-Data Fitted Networks (PFNs; 18) and TabPFN (19; see Section 3.1). PFNs leverage large-scale ML and ICL techniques to approximate Bayesian inference accurately for any prior that can be sampled from. They are trained on millions of synthetic datasets sampled from this prior. For each such dataset, a supervised learning task is constructed, and the model is asked to predict on held out test samples. Then, the PFN is able to apply the principles learned on this synthetic data to real-world datasets, effectively having learned a prediction algorithm. This paper introduces Drift-Resilient TabPFN, an adaptation of the TabPFN framework tailored for tabular datasets exhibiting temporal distribution shifts. Our idea is as follows: Data distribution shifts can be modeled as gradual changes to the structural causal model (SCM; 20, 21) underlying the data. By including the assumption that underlying models change over time into the approximated prior, our models learn to estimate, adapt to, and extrapolate these model changes. While likely no causal model exactly underlies real-world data, we find this approximation to be empirically and intuitively useful. Specifically, we suggest shifting edge weights between nodes that affect causal relationships while sampling instances in each dataset. To accurately model these edge shifts, we propose using a 2 nd -order SCM, a secondary model that adjusts the primary SCM, that takes a time indicator as input and outputs the weight shifts. To represent our temporal domain indicators, we use Time2Vec [22] .A dataset is then a collection of n tuples, denoted as D := {(x i , y i , ĉk )} n i=1 . Each tuple is thereby drawn from a conditional distribution P(X, Y | C = c k ) over the spaces X × Y. Here, c k ∈ C serves as a domain index that conditions the distribution from which the respective sample is drawn. Given that the true underlying temporal domain is mostly unknown in real-world data, it often is approximated and represented as ĉk within the dataset. To isolate samples from a specific domain ĉk , we introduce the sub-dataset D ĉk , defined as D ĉk := {(x, y, ĉ) ∈ D|ĉ = ĉk }. This adapted notation allows for a more nuanced analysis of datasets with domain shifts, building upon the frameworks presented by Wang et al. [26] and Sheth et al. [27] . Domain Generalization (DG). DG aims to train a model that generalizes across a continuum of source domains C train to the target domains C test without access to samples of the target domains. The objective is to learn a mapping function f : X × C → Y that minimizes the expected loss when applied to new, previously unseen target domains.Temporal Domain Generalization (Temporal DG). In temporal DG, a special case of DG, the domain index set C is one-dimensional and follows a total ordering, c 1 ≤ c 2 ≤ . . . [12] . In this framework, the training set is limited to source domains that precede target domains in this ordering, namely C train = {c 1 , c 2 , . . . , c t }. In this setting, the objective is to learn a predictive model f that performs well on these source domains while also robustly generalizing to future, unseen domains C test = {c t+1 , c t+2 , . . . , c n }. The indices of all training domains C train and the index of the current testing domain c k ∈ C test are provided to the model. 
related work: 
 While DG has drawn increasing attention in the research community [28] [29] [30] [31] [32] [33] [34] [35] [36] [37] [38] , its temporal variant remains under-explored. We review existing DG benchmarks on tabular data, DG methods, the temporal DG benchmark Wild-Time [11] , temporal DG methods, as well as relevant TabPFN studies.Tabular Distribution Shift Benchmarks. Several benchmarks have been introduced to assess methods for tabular data under distribution shifts. (1) Shifts and Shifts 2.0 [39, 40] are uncertainty focused benchmarks. Shifts 2.0 includes five tasks, two of which involve tabular data subject to temporal and spatio-temporal shifts. (2) WhyShift [41] , focusing on spatio-temporal shifts, offers five real-world tabular datasets, including the ACS dataset [42] , which we also use in a subsampled form. Evaluating 22 methods like Gradient Boosted Decision Trees (GBDTs), MLPs, and robustness techniques, WhyShift finds that robustness methods do not consistently enhance out-of-distribution (OOD) performance. They also find that while GBDTs perform better, the gap between in-distribution (ID) and OOD performance persists, likely due to GBDTs being better fitted to the ID distribution.(3) TableShift [14] includes 15 tabular binary classification tasks, with 10 relevant to DG. Their work evaluates 19 model types, including several DG methods, finding that methods tailored for distribution shifts do not consistently outperform GBDTs. While generalization gaps can be slightly reduced, each robustness method comes at the cost of ID performance. (4) Wild-Tab [43] focuses on domain generalization within tabular regression using three large datasets. Their study compares 10 generalization techniques against standard Empirical Risk Minimization (ERM) applied to MLPs. Similar to previous work [44] , they find that ERM was not consistently outperformed by specialized DG methods. Notably, no advantage of GBDTs over ERM on MLPs was observed in their datasets. Unlike these benchmarks, which focus on large-scale datasets, our work addresses the overlooked challenges of small-scale temporal distribution shift datasets. However, their insights on generalization and robustness methods align with our findings, providing valuable context for our approach. DG Methods. Several techniques have been proposed to improve robustness to distribution shifts in DG [45] . Key approaches include domain-invariant learning methods, such as Deep CORAL [35] , IRM [31] , and DANN [36] , which aim to learn representations that generalize across domains. Data augmentation strategies, like Mixup [33] and LISA [34] , contribute to generalization by generating synthetic data variations. Additionally, robust optimization techniques, including VRex [37] , GroupDRO [32] , EQRM [38] , and SWA [46] , aim to improve performance under distributional shifts by optimizing for worst-case scenarios or incorporating model uncertainty.In relation to our work, data augmentation strategies are conceptually similar, as we also teach the model DG rather than adding invariances to the architecture. In contrast to augmentation, though, we learn to generalize using completely artificial data on the meta level rather than through manipulation of the target dataset. The other methods focus on designing models specifically for DG, while our approach completely relies on the model to learn to handle distribution shifts. While similar to continual meta-learning (CML) approaches, our method uses domain identifiers to generalize to unseen domains, whereas CML continuously adapts to evolving contexts without identifiers [47] .Wild-Time Benchmark. Wild-Time [11] is a benchmark of five datasets designed to study the real-world effects of temporal distribution shifts -an area largely overlooked by previous benchmarks. While Wild-Time primarily uses non-tabular data, it evaluates a wide array of techniques on its tabular dataset, including classical supervised learning (ERM), fine-tuning, and several previously mentioned general DG methods adapted to temporal distribution shifts. Despite the diversity of methods evaluated, Wild-Time reveals a significant performance gap between ID and OOD data, with none of the 13 tested methods consistently outperforming the standard ERM approach.In our evaluation, we employ their evaluation strategy Eval-Fix and also benchmark our approach against the methods they considered. An in-depth overview of these methods is given in Appendix A.6.Temporal DG methods. Recent specialized temporal DG methods include: (1) DRAIN [12] , which employs a Bayesian framework alongside a recurrent neural network for predicting the dynamics of the model parameters across temporal domains. (2) GI [13] , which explicitly incorporates the temporal domain as a feature, using a specialized Gradient Interpolation loss function, a time-sensitive activation, and enhanced domain reasoning via Time2Vec preprocessing [22] . However, both DRAIN and GI are limited in their ability to extrapolate beyond the near future, leading to their exclusion from our main evaluation. Notably, on the Rotated Two Moons Dataset -where DRAIN and GI have demonstrated their capabilities -our approach outperforms both. See Appendix A.4.3.4 for details.Recent TabPFN Studies. To overcome the limitations of TabPFN in handling large samples and feature sets, typically found in DG benchmarks, several improvements have been proposed. Ma et al. [48] introduced a data distillation approach, where a distilled dataset serves as the model's context, optimized to maximize the training-data likelihood. Similarly, TuneTables [49] , uses prompt-tuning to compress large datasets into a smaller context. Either of these methods could potentially be combined with Drift-Resilient TabPFN, as our modifications focus primarily on the pre-training phase, which remains unchanged in their approaches.Another TabPFN variation, ForecastPFN [50] , also introduces time dependence, but it does not consider any features. It only models simple time series data. Unlike our approach, which builds on TabPFN's SCM architecture for pre-training to handle a large set of features, ForecastPFN models synthetic time series using a single handcrafted function with sampled hyperparameters, simplifying the architecture while trading off diversity of the synthetic datasets. 
methodology: 
 Our approach is built on PFNs, which use ICL to learn the learning algorithm itself. This approach also has a theoretical foundation as described by Müller et al. [18] : It can be viewed as approximating Bayesian prediction for a prior defined by the synthetic datasets. The trained PFN will approximate the posterior predictive distribution (PPD) and thus return a Bayesian prediction for the specified distribution over artificial datasets used during PFN training.Hollmann et al. [19] introduce a prior based on Structural Causal Models (SCMs; 20; 21) to model complex feature dependencies and potential causal mechanisms underlying tabular data. To sample one dataset, this prior samples an SCM which is then used to sample the examples in the dataset. In this approach, each causal representation of a sampled SCM is converted into a functional representation to enable forward computation and dataset sampling.Consider a sampled SCM graph G = (Z, R), where Z = {z 1 , . . . , z n } are the nodes (mechanisms) and R = {r 1 , . . . , r m } the edges (relationships). Each node is set using an individual assignment function f i , z i = f i ({z j | j ∈ PA i }, ϵ i ) , with PA i being the parent nodes of z i and ϵ i random noise.In the following, we will detail the underlying functional graph representation G of an SCM G sampled from the prior. We will later on apply distribution shifts on it. This representation is one level below the SCM and all nodes represent a single scalar value and all edges correspond to linear connections. The values of each node v in G are set just like neurons in a neural network as a simple weighted sum and an activation function v := h j v ′ ∈PAv w i,j v, ϵ j , where the scalar weights w i,j and the activation function h j are randomly chosen at graph creation and the noise term ϵ j is sampled in each forward propagation to create a variety of samples within a dataset.The functional graph representation G of an SCM G is defined as follows:1. Node Expansion. Each node z i is expanded into a set of subnodes Z i := {z 1 i , . . . , zk i }, each representing a scalar value. 2. Graph Expansion. The following steps are performed for each node z j with a non-empty parent set PA j :(a) A new set of subnodesF j := { f 1 j , . . . , f l j } is added. (b) We add the edges E j := {Z i | i ∈ P A j } × F j ∪ F j × Z j .Finally, the functional graph representation is defined asG := ( i∈I Z i ∪ j∈I F j , i∈I E i ).The features of our dataset are defined by a random subset X ⊆ i∈I Z i and the target is defined by a randomly chosen y ∈ i∈I Z i \X. Samples in a dataset are now generated by (i) sampling all noise terms ϵ j in G, (ii) propagating the values of all nodes through the graph, and finally (iii) retrieving values at the feature and target nodes. The resulting datasets, thus abide the computational flow of both G and G. For a more intuitive understanding of this construction, refer to the simplified example provided in Figure 2 . Shaded nodes indicate that their activations cannot be sampled. Feature nodes are blue, the target node is green, input/noise nodes are purple, and all others are gray. The figure also shows the mapping of shifted edges between a causal relationship and its functional form in red, ensuring that shifts specifically target the intended causal relationships without affecting others.We extend TabPFN's prior to model distribution shifts, allowing the model to expand its posterior predictive distribution (PPD) calculations to incorporate temporal domain information. We propose modeling the dynamic of edge shifts using a 2 nd -order SCM. This 2 nd -order SCM is itself an SCM with feature nodes specifying the magnitude of edge shifts in the base SCM's functional graph. Specifically, to extend the scope of the functional representation of the SCM G, we introduce a version Gc k , depending on the temporal domain c k ∈ C. Our objective is to construct a timedependent dataset D which is an aggregation of individual datasets D c1 , D c2 , . . . , D cn , where each datasetD c k := {(x i , y i , c k )} nc k i=1 contains n c k samples.To do this, we sample the temporal domains C = {c 1 , c 2 , . . . , c n } and for each domain c k , we sample the number of samples n c k it contains. An illustration of exemplary domain distributions across datasets is shown in Figure 14 .We then select a sparse subset of relationships in the causal representation of the SCM G to undergo temporal shifts based on the evidence that sparse shifts allow for causal reasoning [1] . For these selected edges, the 2 nd -order SCM H is used to sample shift parameters that govern the corresponding edges in the functional representation Gc k . See Algorithm 1 for a high-level overview.Employing a 2 nd -order SCM for Edge Shifting. The 2 nd -order SCM H takes temporal domains C as input and, through a single forward pass on the corresponding functional representation H, produces dynamic edge shifts for each edge weight w i,j in G that corresponds to an edge in G that should be shifted. Note that although we perform a forward pass, there is no backward pass associated with it. Each 2 nd -order SCM is randomly generated and used solely for sampling the weight shifts. This design allows for Bayesian reasoning over the edge shifts and enables H to generate complex, often correlated shifts over time. While we emphasize this construction does not reflect the true underlying dynamics of edge shifts in many real-world datasets, we use this heuristic prior construction for the following reasons: (a) Features in the SCM are correlated to each other in varying degrees, mimicking real-world processes of correlated changes in the underlying generating mechanisms of real-world data (b) An SCM with NN based causal mechanisms and nonlinear activation functions can extrapolate values outside of the data distribution, generalizing the underlying function to future domains (c) As demonstrated by TabPFN, a causal model prior is a sufficiently general approximation to many real-world datasets. We have visualized this approach in Figure 3 and a selection of the functions generated by a 2 nd -order SCM in Figure 15 .Shifting SCM edges can model various Types of Distribution Shifts. A distribution shift is characterized by changing distributions between contexts (P(X,Y | C = c i ) ̸ = P(X, Y | C = c k )).This definition can be further broken down into the following scenarios:Covariate Shift: Changes in the feature distribution (P(X)) while the conditional label distribution (P(Y |X)) remains constant.Prior Probability Shift: Changes in the label distribution (P(Y )) while the feature distribution given the labels (P(X|Y )) remains constant.Concept Shift: Changes in the relationship between features and labels (P(Y |X) or P(X|Y )) while the marginal distributions of features or labels remain constant. Diagram illustrating the integration of a 2 nd -order SCM for adaptive edge shifting across evolving temporal domains. On the right, the primary network G generates data samples over multiple time domains, with red arrows indicating shifted edges. On the left, the 2 nd -order SCM -an auxiliary network H -takes an input domain c k ∈ C and outputs parameters to adaptively shift each edge weight w i in the base network. [51] represented as Bayesian networks as defined by Kull and Flach [52] . Here X, Y , and C denote the random variables of the features, label, and context, respectively. Note that all these types of shifts naturally arise in our prior, since we sample feature and target positions, as well as the locations of shifted edges, randomly at various positions in the synthetic datasets.Y X C (a) Covariate Shift Y X C (b) Prior Probabil- ity Shift Y X C (c) Concept Shift (X → Y ) Y X C (d) Concept Shift (Y → X) Y X C (e) Other Types of ShiftThese shifts can also be viewed as Bayesian networks shown in Figure 4 -all arising in our prior by sampling features and targets at varying SCM positions. For further visualizations, refer to Figure 13 . Temporal domains and features are normalized using only the training data, and so, to effectively encode temporal information and provide normalized inputs when projecting into the future, we use Time2Vec [22] . It converts each temporal domain index into an m-dimensional vector, using linear and sinusoidal functions characterized by learned parameters ω i and φ i . Specifically, the Time2Vec transformation for a temporal index c k ∈ C is formulated as:EQUATION4 ExperimentsEvaluation Strategy. We evaluate analogous to the Eval-Fix setting outlined in Wild-Time [11] , measuring both, ID and OOD performance. Here, each dataset D is split into three subsets: D train , D ID , and D OOD . Splits are based on a randomly sampled temporal domain c k that serves as the boundary between the train and test (OOD) portion. We only use such splits, where D train comprises between 30% and 80% of the total domains and samples. To assess ID performance, we subsample 10% of the instances in each domain of D train as the ID test set and the remainder as the train set. An illustration of the Eval-Fix strategy is provided in Figure 16 .Each class in the training set is required to be represented in both D ID and D OOD , and vice versa. For all datasets, we generate three random splits and average metrics across these splits. We had to limit the number of splits to three due to the constrained number of available domains and the requirement for classes to be present in both the train and test splits. Each method is trained three times, and we report the average and 95%-confidence intervals calculated across model initializations.We evaluate Accuracy, F1-Score (Harmonic mean of precision and recall, useful in imbalanced datasets), ROC AUC (Area under the receiver operating characteristic curve), and ECE (Expected Calibration Error; reflects the reliability of the model's probability outputs).Datasets. Our benchmark comprises 18 test datasets, 8 synthetic and 10 real-world. In addition, 12 validation datasets, 4 synthetic and the remaining 8 real-world, were used to optimize the hyperparameters of our approach via random search. While some of these datasets have been analyzed in previous work, there has been no comprehensive benchmark focusing on small tabular datasets undergoing distribution shifts. To address this gap, we carefully selected and generated a diverse range of datasets that exhibit temporal distribution shifts. The selected datasets originate from open dataset platforms or previous work in DG. Ground truth domain indices c k ∈ C are known for synthetic datasets. For real-world datasets, we approximated domain indices ĉk based on features that encode temporal information, which we transformed into discrete intervals. Also, some real-world datasets required subsampling due to their large size, which was beyond the current architecture of TabPFN. We provide full details, including descriptions for each dataset and pre-processing steps in Section A.7 of the Appendix.Baseline Setup. Our baselines include state-of-the-art methods for tabular prediction. These include advanced GBDTs like CatBoost [24] , XGBoost [23] , and LightGBM [25] , which have demonstrated superior performance to standard neural network approaches in handling tabular data [17] . We also include TabPFN in its unmodified form (TabPFN base ; 19) . Methods from the Wild-Time benchmark are examined separately and detailed in Section A.4.4.3 of the Appendix. However, we have added the two best-performing Wild-Time methods, ERM and Stochastic Weight Averaging (SWA), to the main results table. All baseline methods besides TabPFN are subject to a time budget of 1,200 seconds on 8 CPUs and 1 GPU. For each method except TabPFN, which does not require tuning, a random hyperparameter search with 3-fold time series cross-validation was used. We chose the best-performing hyperparameters based on OOD ROC AUC within the allocated time.Among our baselines, we considered three strategies:1. Providing the full dataset D train along with the corresponding domain indices C train as a feature, aiming to allow for better reasoning of the shifts in the dataset (all dom. w. ind.). i=1 (last dom. wo. ind.). The rationale behind the last scenario is to provide only training data closest to the subsequent test distribution. This strategy is not used for distribution shift baselines.TabPFN Setup. For the TabPFN variants, both the original and our modified method (TabPFN dist ) were pre-trained for 30 epochs across 8 GPUs. This results in a total of 30,720,000 synthetically generated datasets processed during pre-training. While this pre-training step is moderately expensive, it is done offline, in advance, and only once as part of our algorithm development. Furthermore, the preprocessing parameters of both methods were optimized once on the validation datasets by random search over 300 configurations. We chose the configurations that yielded the best OOD ROC AUC performance. The resulting model and hyperparameters are used for all datasets, resulting in, on average, 110 times faster training and prediction time on our benchmark.Quantitative Evaluation. Our method demonstrates superior predictive performance across all metrics for OOD data in 18 test datasets, for synthetic datasets and real-world datasets, as detailed in Table 1 . Compared to the strongest baseline, our method improves accuracy from 0.665 to 0.754 on synthetic datasets and from 0.712 to 0.736 on real-world datasets. It also enhances the F1 score from 0.588 to 0.697 on synthetic datasets and from 0.668 to 0.682 on real-world datasets. Additionally, it increases the ROC AUC from 0.749 to 0.844 on synthetic datasets and from 0.82 to 0.822 on realworld datasets. Furthermore, our method shows much stronger calibration on OOD samples, reducing ECE from 0.164 to 0.126 on synthetic datasets and from 0.083 to 0.062 on real-world datasets. While baselines are often overconfident on OOD data, our method is able to predict uncertainty accurately. Compared to TabPFN and GBDTs, the NN-based methods in the Wild-Time Benchmark (Section A.4.4.3) show a substantial drop in performance, likely due to the limited training data. Since our method focuses on enhancing OOD robustness rather than optimizing ID tasks, we find lower predictive performance on ID tasks. While performance gains are observed on both, real-world and synthetic data, we observe stronger improvements on synthetic datasets. This can be partly attributed to the, on average, stronger distribution shifts between ID and OOD data in our synthetic benchmark. Furthermore, real-world datasets often show multifaceted and complex shifts that are much more difficult to extrapolate into the future. Combined results across all datasets are provided in Table 5 . 7 . Likewise, plots illustrating the overall shifts in these datasets are provided in Figure 6 .  
conclusions & limitations: 
 In this work, we presented a Bayesian approach to address the issue of temporal domain generalization in tabular data. Specifically, we focused on enhancing TabPFN to improve its robustness to temporal distribution shifts. Within this framework, we introduced a novel approach that changes the causal relationships in the SCM prior over time, thereby enabling TabPFN to inherently adapt to these shifts. Our method outperforms all baselines on the evaluated datasets and demonstrates notable improvements both qualitatively and quantitatively, particularly on synthetic OOD datasets. Furthermore, it requires no hyperparameter tuning, is not limited to particular types of distribution shifts and takes only 10.9s for training and prediction combined.Despite these advancements, our methodology inherits certain limitations from the underlying TabPFN model. (1) Due to the quadratic scaling of the attention mechanism with respect to the number of samples, our method does not scale to large datasets. Here, our research will benefit from the continued improvements of TabPFN, ICL, and sequence-based models in general.(2) The TabPFN, like many transformer-based models, acts as a "black box", making it challenging to interpret the model's predictions and understand the recognized distribution shifts. (3) The underlying prior for structural causal models with sparse mechanism shifts may not accurately describe all real-world datasets. While we find it to be empirically and intuitively useful, real-world shifts might have underlying complexities that our prior currently does not capture.For future work, next to addressing the existing limitations, we have identified in initial experiments promise in extending our model to (1) transductive and (2) online continual learning settings. Also, our prior could be adapted to support (3) spatial or spatio-temporal distribution shifts. Employing a prior that (4) models shifts in the underlying causal model could improve the robustness of the baseline TabPFN in standard classification tasks where temporal shifts are often implicit.We provide code, pre-trained models, and a Colab notebook at https://github.com/automl/ Drift-Resilient_TabPFN. We further describe reproducibility, the release of code and models, the broader impact of our models, and the computational resources used for method development in Appendix A.1, A.2 and A.3. We add a discussion of baselines in A.6, an in-detail discussion of the evaluated datasets in A.7, and additional quantitative and qualitative evaluations in A.4. 
acknowledgments: 
 Frank Hutter acknowledges the financial support of the Hector Foundation. This research was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant number 417962828.We acknowledge funding by the European Union (via ERC Consolidator Grant DeepLearning 2.0, grant no. 101045765). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.included in the submission code. Code to generate our synthetic datasets can be found in our code repository, see above.Details of Hyperparameters for Drift-Resilient TabPFN and Baselines. An overview of the hyperparameters used for running Drift-Resilient TabPFN and our baselines can be found in Tables 8, 9 and 10 respectively.Evaluation Reproducibility. Consistent dataset splits were pre-determined to ensure comparability across all evaluations, reinforcing the reliability of our findings. 
 
 Appendix: As a general method for handling distribution shift in tabular data, Drift-Resilient TabPFN does not have immediate direct societal implications in the same way as AI systems designed to automate specific tasks or replace human jobs. However, Drift-Resilient TabPFN could offer several potential positive societal impact:1. Increased Model Longevity: Our approach extends the usable lifespan of deployed ML models by adapting to distribution shifts, reducing the need for retraining and leading to cost savings and more stable performance. In healthcare, this can ensure diagnostic and prognostic models remain reliable as data shifts over time. 2. Improved Decision Making and Long-Term Predictions: Drift-Resilient TabPFN enhances decision-making in critical fields like finance and climate science by enabling more robust, longer-term predictions. Our Bayesian approach for tackling distribution shift provides a new perspective that can spur further methodological innovations.However, we also recognize potential negative impacts:1. Potential Misuse: Like any ML advance, more robust models could be misused for harmful purposes if not developed and deployed responsibly. 2. Environmental Cost: While the trained model is now efficiently applicable to various datasets, the considerable computational resources used for initial training should be noted. However, we note that these costs are one-time, while the resulting model can be applied with minimal energy usage.In the course of our research on Drift-Resilient TabPFN, we employed substantial computational resources across various stages of model development, training, and evaluation. The computation specifics are as follows:1. Infrastructure: The experiments were conducted on an internal SLURM cluster equipped with RTX 2080 TI GPUs and CPUs of type AMD EPYC 7502, 32C/64T, @ 2.50-3.35GHz. Additional computational resources were allocated for method development tests and other experimental setups not detailed in the final publication. Thus the full scope of the research required more computational resources than those detailed above due to these preliminary and unreported experiments.Is our model's performance mostly based on Time2Vec preprocessing? To address this question, we conducted an ablation study where we trained a model with temporal domain indices normalized but not subjected to Time2Vec preprocessing (No T2V). This additional experiment analyzes the impact of perturbing temporal domain indices on the performance of our Drift-Resilient TabPFN model. In real-world datasets, ground truth temporal domain information is often unknown and must be approximated, creating a crucial difference between these datasets and those generated by the prior in our methodology. To assess this, we conducted an experiment during model development wherein we intentionally modified the ground truth domain indices C during the pre-training phase prior to feeding them into the transformer.In this context, we explored three primary techniques:Shifting Domain Boundaries Probabilistically. We shifted the boundaries of domains and reported instances near those boundaries as belonging to adjacent domains.Merging Domains. Multiple domains were probabilistically merged into a single domain, thereby introducing ambiguity in domain information.Noise Injection. Random noise was added to each domain indicator, further complicating reasoning about temporal distribution shifts based on these indicators.Our experiments on our validation datasets, listed in Table 3 , show that overall these perturbations adversely affect model performance. The findings suggest that the model requires ground-truth domain indices for effective training, emphasizing the importance of accurate domain information in real-world applications. To show our improved performance compared to the state-of-the-art methods DRAIN [12] and GI [13] , we compare our method with the qualitative analysis performed by the authors of DRAIN on the Rotated Two Moons dataset. In this setting, all domains except the last are used for training, with the final domain reserved for testing. We illustrate our method's decision boundary compared to those provided by DRAIN in Figure 9 . The accuracy results are listed in Table 4 . As the contour levels are unknown, we display only the pure decision boundary for clearer comparison. Our analysis shows that our model forecasts the rotation of the two moons more accurately compared to the baselines and adapts its decision boundary more precisely.Train Domains: 0-8 | Test Domain 9 This section offers a quantitative evaluation of our model against methods found in the Wild-Time benchmark [11] . We focus exclusively on methods applicable to tabular data, omitting SimCLR and SwAV which are specifically designed for image datasets. Detailed explanations of these methods are available in Section A.6.As a base model, we used a Multilayer Perceptron (MLP) optimized through Hyperparameter Optimization (HPO).The findings of this quantitative comparison are presented in Table 6 . Notably, none of the evaluated Wild-Time methods demonstrated performance equal with our approach or the other baseline methods on OOD data. This discrepancy is likely due to the small number of instances in the datasets used in our evaluations, which affects the generalization of these deep learning techniques. Among the Wild-Time methods, SWA emerged as the most effective in handling OOD data. In this subsection, we assess the computational budget allocated for HPO on our main baseline methods. By comparing model performance at 1200 seconds versus 3600 seconds per method and dataset split, we show that the increased budget for HPO does not yield significant gains overall, indicating sufficient saturation within the initial budget. In this supplementary evaluation, we investigate the correlation between the inherent difficulty of a dataset concerning distribution shifts and the performance of our approach in comparison to the baselines.Defining Difficulty Metrics. Our first step involves assigning a difficulty score to individual dataset splits, using a basic XGBoost model that doesn't include domain indices as a feature of the dataset. This score is determined using the formula ID -OOD, where both ID and OOD performances are measured using ROC AUC. This difficulty score quantifies the decline in model performance when transitioning from ID to OOD data. After calculating the difficulty scores for each dataset split, we proceed to train each of the evaluated methods across all domains, including the domain indices. We then compute the performance gap, which is also calculated as ID -OOD, for each method.These metrics are then visualized by plotting the dataset's difficulty score against its corresponding performance gap for each dataset split. We also fit linear regression lines to the scatter plots representing each method. The final plot is illustrated in Figure 11 .Although the data points cannot be interpreted directly, the linear regression suggests that our method experiences the least decline in ROC AUC performance as dataset difficulty increases, affirming its robustness. The TabPFN baseline is the second-best performer, while the other models show more significant drops in performance. For this analysis, we evaluated the performance of both our method and TabPFN-base by fixing the prediction to the last domain of the Intersecting Blobs dataset and gradually increasing the number of domains available during training. Our results confirm observations made during method development: Our method requires significantly fewer training domains to accurately extrapolate shifts into the distant future, while the TabPFN-base only has acceptable decision boundaries when there is data whose distribution is close to the test domain. Below in Figure 12 are the results for this experiment with the Intersecting Blobs dataset discussed in Section A.4.3.1. The figure clearly shows that our model greatly improves its predictions with the first domains. When provided with the domains C train = {0, . . . , 3}, the shift is largely understood, with newer domains having little impact on performance. On the other hand, TabPFN-base requires much more training domains to achieve similar performance to our method. Algorithm 1 This algorithm provides a high-level overview for generating a synthetic dataset in our prior. Although steps are depicted sequentially for clarity, many can be parallelized in actual implementation. D c k ← ∅ ▷ Initialize sub-dataset 12:for all i ∈ {1, ..., n c k } do ▷ Sample sub-dataset 13:(x i , y i , c k ) ← Gc k .FORWARD(ϵ i ) 14: D c k ← D c k ∪ {(x i , y i , c k )} 15:end for 16:D ← D ∪ D c k ▷ Extend dataset 17:end for 18:return D ▷ Return dataset 19: end procedureIn Figure 16 , we provide an illustration of the Eval-Fix evaluation strategy, originally proposed by Yao et al. [11] in the context of the Wild-Time benchmark. It should be noted that the formalism has been adapted to align with the notation used in this paper. Swapping Assignments between multiple Views of the same image (SwAV). SwAV [62] employs a clustering approach within the contrastive learning framework. It enforces consistency between cluster assignments across different augmentations of the same image. This obviates the need for pairwise feature comparisons, offering computational efficiency.A.6.5 Bayesian Learning Stochastic Weight Averaging (SWA). SWA [46] averages multiple parameter values along the stochastic gradient descent (SGD) trajectory to improve in-distribution generalization. It operates with minimal computational overhead and aims to approximate the posterior distribution over model parameters, reflected in the flatness of the learned optima. [11] A. The task involves predicting mortality based on 9 features, which include age, sex, year of blood sample, FLC portions (kappa and lambda), the FLC group, serum creatinine, MGUS diagnosis, and days from enrollment to death or last follow-up. The feature 'chapter' was omitted because it is direct information on whether someone died or not. We treat "sample.yr", the year in which the sample was taken, as a domain, resulting in a total number of 9 domains. We suspect that both the measurements themselves and the selection of participants have changed over time. The dataset is sourced from studies by Dispenzieri et al. [72] and Kyle et al. [73] . Dataset 22 (Electricity). This dataset as used by Harries [74] and Gama et al. [75] contains electricity demand in the Australian New South Wales Electricity Market. Since the prices are not fixed, they fluctuate depending on supply and demand. It has 45,312 instances and 5 features. For reproducibility, the dataset includes additional features such as the New South Wales electricity price, which was used to form the target class according to the original paper, and the Victoria electricity price, which was not used in the original paper. The dataset features the demand of electricity in to provinces as well as the transfer between those for periods of 30 minutes. The task is a binary classification, which requires predicting whether the price in the current period is higher or lower than the average of the last 24 hours. The dataset contains seasonal data due to varying demand for electricity. The effects of long-term price trends on the class label are removed by the 24-hour moving average. We consider one-week periods as a single domain. To comply with TabPFN sequence length limits, we keep only two hourly intervals for each day and subsample 15 weeks of the whole time period. Dataset 23 (Absenteeism at Work). The Absenteeism at Work dataset, sourced from the UCI Machine Learning Repository provided by Martiniano and Ferreira [76] , comprises 740 instances across 21 features. It captures various attributes of employees and their working conditions, such as the reason for absence, day of the week, seasons, distance to work, and more, with the target feature being the absenteeism time in hours which was discredited into 4-quantiles.The primary shift in this dataset is supposed to be seasonal. Thereby each consecutive season is treated as a different domain. Furthermore, no significant preprocessing or subsampling was required due to the manageable size of the dataset. Dataset 24 (Heart Disease Dataset). The Heart Disease dataset, sourced from the UCI Machine Learning Repository and provided by Janosi et al. [77] , targets the classification task of identifying the presence (values 1,2,3,4) or absence (value 0) of heart disease in patients. We treat the task as a binary classification, predicting only whether a patient has heart disease or not. It includes 303 instances, each with 13 health-related features such as age, sex, resting blood pressure, cholesterol levels, etc. In this context, each consecutive 4-year age interval is viewed as a single domain. Rows with missing values have been omitted. Dataset 25 (Parking Birmingham). The Parking Birmingham dataset, provided by Stolfi [78] via the UCI Machine Learning Repository, initially comprises the capacity and occupancy rates(target) of multiple car parks. We have processed it to only include the car park labeled 'Others-CCCPS133', thereby reducing the number of instances to 1,294 from 35,717. The original 'LastUpdated' attribute has been transformed into 'day', 'week', and 'month' features, with the 'week' serving as the temporal domain. The 'Occupancy' target, originally an absolute figure, is now presented as a percentage of the parking space utilization, discretized into 25% intervals. Dataset 26 (Ames Housing Prices). The Ames Housing Dataset, curated by De Cock [79] , consists of 1460 instances each with 79 features detailing various aspects of residential homes in Ames, Iowa. We use only the training portion of this dataset due to the absence of ground truth targets in the test data. We have discretized the house price, which was originally a continuous variable, into a categorical variable. This transformation was achieved by partitioning the price data into intervals. Specifically, the intervals are defined as: [0, 125k], (125k, 300k], (300k, ∞). The task is to predict the price range of a home based on its features. We treat the 'YearBuilt' attribute, divided into 15-year periods, as our domain to capture changes in housing trends over time. It is to be expected that the data set shows temporal shifts, as the price distribution between older and more modern houses differs.Dataset 27 (Folktables US Census). The folktables datasets, derived from the US Census Public Use Microdata Sample (PUMS) data and published by Ding et al. [42] consists of demographic and socioeconomic data between 2015 and 2021. Each year within this timeframe represents a distinct domain for a series of tasks: ACSIncome, ACSPublicCoverage, and ACSEmployment. We purposefully limited our focus to the state Maryland, to limit the shifts to the temporal domain.The size of the datasets necessitated a stratified subsample for the target per year to reach a total of approximately 1300 instances per dataset and meet the TabPFN model requirements. This subsampling ensured a representative yet computationally manageable sample.The tasks are as follows:• ACSIncome: The task predicts whether an individual's income surpasses $50,000, narrowing the ACS PUMS data to individuals over 16 who reported at least one working hour per week in the past year and a minimum income of $100. The task consists of 10 features accross the 7 domains.• ACSPublicCoverage: The objective is to predict if an individual is covered by public health insurance. The dataset is filtered to include only individuals under 65 with an income below $30,000, focusing the prediction on low-income individuals ineligible for Medicare. The task consists of 19 features accross the 7 domains.• ACSEmployment: The task is to predict whether an individual is employed. The dataset is filtered to include only individuals between 16 and 90 years of age. The task consists of 16 features accross the 7 domains. Dataset 28 (Chess). The Chess dataset published by Žliobaitė [80] is derived from recorded chess games, aiming to predict game outcomes (draw, lost, won) through a multi-class classification task. It consists of nine features which provide insights into the game details and player attributes, including move sequences, player side (white or black), current rating, opponent's rating, type of game, speed, and the date of the game (broken down into year, month, and day). The dataset is segmented into 27 domains, where each domain represents 20 consecutive games. This segmentation helps capture the evolution of a player's progress over time. The dataset, in its entirety, holds 533 instances. It has been constructed based on games played between 7 December 2007 and 26 March 2010. 38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: This paper extends Prior-Data Fitted Networks (PFNs) for tabular data (TabPFNs) to handle temporal distribution shifts by augmenting the base model with a structural causal model (SCM) as a prior.  The core idea is to use a hypernetwork (which the authors later call a 2nd-order SCM) to perform amortized computation to adjust the SCM to the drift.  The reviewers were unanimously in favor of acceptance albeit with varying degrees of enthusiasm (1 x borderline accept, 2 x weak accept, 1 x accept).  Reviewer aHHC had the most negative opinion of the work, with the primary criticisms being the lack of clarity in the description of the training process and the work being incremental in nature.  The former issue was cleared up the authors in the rebuttal.  The latter issue still remained, but in my judgement, the paper is a clear extension of TabPFNs, and while the novelty lies in including a standard technique from the SCM literature, the exact methodology does not yet exist and demonstrates empirical merits.  

Reviewer LELq took issue with the paper's framing and presentation of related work (which was echoed in other reviews).  In my own reading, indeed the current draft suffers from some over generalizations (e.g. scoping all shifts vs a focus on temporal, "Until now, no method has significantly outperformed classical supervised learning...") and a lack of organization in its description of related work.  The related work, while there is a dedicated section, is still dispersed throughout the introduction and even in Section 3.  The authors have promised to address these issues in the camera-ready version.  While the authors should certainly make these changes, I feel that these are mostly cosmetic issues that do not jeopardize the core contribution of the work.

Reviews: review: paper_summary: paper_summary: This paper studies the setting of non-iid train / test distribution shift in the tabular machine learning. The authors extend the previous TabPFN sota tabular model to deal with domain shift cases. TabPFN uses SCM to model data prior, and in this paper, the core idea is to update the SCM prior graph by hypernetworks. The output of hypernetworks is the SCM update. As a result, TabPFN could deal with data shift. The whole idea is very simple and the authors demonstrate the superior effectiveness in various synthetic and real datasets
strength_weakness: strength_weakness: The main weakness lies in the experimentation. See the questions below. Technically, the core part is the integration of hypernetworks, which seems very simple. However, in the experiments, there is little discussion of this module. Besides, since many synthetic datasets are used, it is nature to study the SCM from a more explicit way and especially how hypernetworks work well in the setting.
questions: questions: - If I understand correctly, the core part of the method lies in training a hypernetworks to update SCM. All the rest follows TabPFN?  
- It is vague how hypernetworks is trained and processed. It is suggested to provide details on this
- The authors mentioned in Figure 4, various cases of distribution shifts, how are these shifts mitigated in the experiments? Any investigation on different cases?
- How good has the hypernetworks been trained? Since there is many synthetic datasets used, it is great to study cases where hypernetworks updated the SCM correctly
- How much data is required to train hypernetworks? 
- How is the hypernetworks training related to number of features, number of classes, etc.  since TabPFN has limitations on the dataset requirement.
- In Table 1, it is recommended to separate real datasets and synthetic datasets. It is also necessary to briefly discuss the nature of the used datasets
limitations: limitations: NA
paper_summary: paper_summary: The paper presents a modification of TabPFN that incorporates a novel prior to incorporate temporal shifts. In particular, the proposed method introduces an additional SCM that, through a temporal representation (Time2Vec), learns to modify the model parameters in response to shifts. The authors train a version of the proposed model and compare it to TabPFN, GBDT baselines, and standard ERM and SWA baselines.

Overall, the paper's presentation is good (but could be improved), and this appears to be a logical addition to the TabPFN method. A method capable of zero-shot transfer to new distributions seems especially well-motivated, given that no labeled data from the target domain is available in many real-world scenarios. However, some revisions and additions to the paper are badly needed. For example, the literature review is inadequate and needs a complete rewrite; there are no domain generalization or robustness methods included in the experimental results (besides the proposed method), and the ablation study is too limited (in fact, the current results suggest that there is no advantage to the Time2Vec addition, which raises questions about the overall approach). I think that with extensiv revisions this paper could be brought up to acceptance level and am open to raising my score, but the scope of additions and revisions feels on the borderline of too much for a simple camera-ready phase.
strength_weakness: strength_weakness: See above.
questions: questions: See above.
limitations: limitations: Yes.
paper_summary: paper_summary: The authors propose an extension of the TabPFN framework such that the model can be resilient to distribution shifts during the inference phase by observing the shifted samples in-context. Akin to TapPFN, the pipeline involves pretraining based on an SCM, and the authors introduce a temporal aspect to the SCM construction such that the model will be aware that several common types of distribution shifts can be possible at inference time.
strength_weakness: strength_weakness: I did not feel the work had any major structural weaknesses within the scope of what the authors wished to achieve.

A small suggestion that the title should imply that the work is about temporal domain shifts.

The “runs in seconds” clearly would not be true for all kinds of target scales, although it rightfully highlights the potential of the approach in some scenarios. I recommend adding a short qualifier on the scale where this is true.

Additional clarity on what exactly is the proposed input and output to the final model would have been appreciated in the methods section. For instance, does the model need to observe all previous intervals eg. [0 1 2] and the future intervals [3 4 5]? Only the previous intervals? Or just a trailing list of recent intervals? What is the justification that future intervals [4 5] will be available to the user if they are still in the present (domain 3)? 

Due to some of the ambiguity, it was also not clear at a high level if a user needs to directly indicate to the model that a domain shift is generally expected at certain domain indices, or the model makes such a decision for every domain index internally.

“We approximated domain indices … based on features that encode temporal information, which we transformed into discrete intervals.” It seems that the accuracy attained by the models could vary a bit based on the way that the domains are approximated, unless the discretization was completely random uniform and guaranteed to be fair. Some additional assurance seems deserved on how this approximation was done.

I think the authors motivate the problem enough such that readers will be convinced that existing methods would not do well under distribution shifts. Given this, I think a more competitive benchmark could be to see how well baselines trained on small parts of the OOD do well on the OOD domains, and how this compares with the various ways of applying DSTabPFN. I would be curious to know if this was considered.
questions: questions: Do the authors have any more considerations on the extent to which in-domain (ID) capabilities are affected when adopting the Distribution Shift TabPFN approach as opposed to the original TabPFN? This is hinted in line 287 but it seemed like an important point that gets at some quantitative and motivation-related questions. For instance, what is the false positive rate of the DSTabPFN guessing a distribution shift when there is none? What if there are some outliers in the inference phase despite the underlying distribution will continue longer term? Should users rely on the model to distinguish between one shift to another or only turn to DSTabPFN when a distribution shift is noticed? Some more intuition on this matter may convince readers further.

What is the rationale for the compute budget imposed in line 262? Generally eg. XGB would be parallelized with dozens of cores and requires a long time to cross validate & fit, but perhaps this was not necessary for the scale of the datasets explored.

It would be great to understand how the approach scales with the amount of samples from the SCM. For instance, are x5 more samples (taking x5 times longer to train) needed for the model to be robust to 5 different types of distribution shifts? Or is a similar accuracy to TabPFN possible with an equal number of samples while also being robust to shifts? Is the reason why DSTabPFN does not match the accuracy of TabPFN because it crosses into an overfit regime with too many samples? 

A related analysis that could have been nice to have is to see how many “shots” in the shifted distribution are needed for DSTabPFN to adjust appropriately.
limitations: limitations: The work discusses limitations to a fair extent in the final section. I did not assess that the work could have a major negative societal impact.
paper_summary: paper_summary: This paper aims to address the temporal distribution shifts in Tabular data based on TabPFN. Concretely, the proposed method use two structural causal models (SCM) to model the prior for TabPFN, one for the gradual shift of inductive bias over time and the other is to model the shift of the first SCM. Empirical studies on various datasets illustrate the effectiveness of the proposed approach.
strength_weakness: strength_weakness: 1. As far as I am concerned, CatBoost, XGboost, and LightGBM do not explicitly consider the temporal shift. The proposed method introduced additional sampled data from a temporal shift prior to PFN. I would doubt such a comparison is not that fair. Does any of the baselines in [1]  outperform the proposed method on the datasets used in this paper?
2. The experimental setting could be more detailed in the main paper.
3. It seems the proposed method is a simple modification to TabPFN that changes the SGM prior with additional temporal shift add-ons. So it's a little bit combinatorial to me, but not a huge problem.
4. Modelling the temporal shifts of inductive bias shares similarity with some continual learning approaches using meta-learning techniques, which lack a brief discussion, such as [2, 3, 4].

[1] Benchmarking distribution shift in tabular data with tableshift. Advances in Neural Information Processing Systems,36 (2024).

[2] Online fast adaptation and knowledge accumulation (osaka): a new approach to continual learning. Advances in Neural Information Processing Systems, 33:16532–16545, 2020.

[3] Reconciling meta-learning and continual learning with online mixtures of tasks. Advances in Neural Information Processing Systems, 32, 2019.

[4] On the stability-plasticity dilemma in continual meta-learning: theory and algorithm. Advances in Neural Information Processing Systems, 36 (2024).
questions: questions: 1. Why do you need the functional representation graph $\tilde{\mathcal{G}}$? What are the benefits of the original sampled graph?
2. Why do sparse shifts allow for causal reasoning (Line 195)? 
3. How are the edge weights decided?
4. Why can SCM with NN-based causal mechanisms and nonlinear activations extrapolate values out of data distribution (Line 208)? Is this due to a causal mechanism or NN?
limitations: limitations: Yes.

Comments: title: Follow-up on Reviewer's Response
Score 4 to 6
Follow-up on Reviewer's Response + Extended T2V Ablation (Part 2)
Follow-up on Reviewer's Response + Extended T2V Ablation (Part 1)
Follow up to reviewer response
Additional Experiments and Follow-Up to Reviewer Response
Clarification and Follow-Up on Baseline Evaluation in Response to Reviewer Feedback
Follow up to reviewer response (Part 2)
Follow up to reviewer response (Part 1)
Related Work (Part 3)
Related Work (Part 2)
Related Work (Part 1)
New Version of the Section "Related work"
Follow up to author response
Rebuttal by Authors (contd.)
Rebuttal by Authors (contd.)
Rebuttal by Authors (Part 3)
Rebuttal by Authors (Part 2)
Rebuttal by Authors (contd.)
comment: Dear Reviewer m2Lp,

Thank you for your feedback and for raising your score in light of our revisions. We appreciate your recognition of the improvements we've made and your suggestions for further enhancing the clarity of our paper.

> 1. Add some examples for how to span the nodes and graph, that is, how you create $\tilde{z}_1^1$ and $\tilde{z}_1^2$ from $z_1$ and how you $\tilde{f}_2^1$. The current presentation in Section 3.1 is very obscure.

We thank you for this suggestion and will include these details in the camera-ready version to make the methodology more accessible to the reader.

> 2. Move some important experimental results and details to the main paper.

We are glad that you see the value of the additional experimental results that we have included in the appendix, and acknowledge that some of these results need to be highlighted more prominently in the main paper. 

As mentioned in our latest response to Reviewer LELq, we move a short discussion on the key findings related to the Wild-Time baselines as well as the results of the best performing DG method in Table 6, into the main paper. We also add a table of contents to the beginning of the appendix to provide an easier overview of the additional experiments included. In addition, we summarize the main results of the Time2Vec ablation in Section 4 - Impact of Time2Vec Preprocessing on our Model Performance (L305 - L307). Furthermore, we split Table 1 to present synthetic and real-world results separately, as previously done in the appendix, with corresponding discussions in Section 4 - Quantitative Results. 

We believe these changes provide more clarity to the reader and offer a better picture of the robustness of our approach. However, if there is anything beyond these changes that you believe should be included in the main text, we would be glad to consider those as well.

> 3. Add discussion w.r.t related works and other explanations or verifications to your claims, as presented in the rebuttal.

We appreciate that you found the revision of the related works section, as well as the other proposed clarifications and additional experiments beneficial. We will include these in the camera-ready version.

Thank you once again for your feedback and for acknowledging our efforts to refine the paper.

Best regards, The Authors
Thanks for the authors' detailed reply. 

Most of my concerns are addressed. I raised the score accordingly. 

Based on your response, I recommend the following changes to improve the paper's clarity:
1. Add some examples for how to span the nodes and graph, that is, how you create $\tilde{z}_1^1$ and $\tilde{z}_1^2$ from $z_1$ and how you $\tilde{f}_2^1$. The current presentation in Section 3.1 is very obscure.

2. Move some important experimental results and details to the main paper. 

3. Add discussion w.r.t related works and other explanations or verifications to your claims, as presented in the rebuttal.
In addition, we are pleased to now provide the results from the additional iterations of the no-T2V ablation, which are presented in the table below. This table is an extension of Table 2 from our initial submission, now including the mean and confidence intervals across three initializations for the no-T2V variant.

| **Model**                | **Variant**           | **Acc. ↑ (OOD)**           | **Acc. ↑ (ID)**            | **F1 ↑ (OOD)**             | **F1 ↑ (ID)**              | **ROC ↑ (OOD)**            | **ROC ↑ (ID)**             | **ECE ↓ (OOD)**            | **ECE ↓ (ID)**             |
|--------------------------|-----------------------|----------------------------|----------------------------|----------------------------|----------------------------|----------------------------|----------------------------|----------------------------|----------------------------|
| **TabPFN-dist** | all dom. w. ind.      | **0.744** (.018)           | 0.879 (.012)               | **0.689** (.028)           | 0.837 (.022)               | **0.832** (.018)           | 0.932 (.002)               | **0.091** (.006)           | 0.074 (.014)               |
| **No T2V**               | all dom. w. ind.      | 0.742 (.004)               | 0.877 (.007)               | 0.685 (.002)               | 0.834 (.014)               | **0.832** (.004)           | 0.931 (.009)               | 0.093 (.009)               | 0.071 (.005)               |
| **TabPFN-base** | all dom. w. ind.      | 0.688 (.010)               | **0.885** (.010)           | 0.620 (.012)               | **0.847** (.017)           | 0.786 (.007)               | **0.935** (.010)           | 0.119 (.006)               | **0.067** (.005)           |
|                          | all dom. wo. ind.     | 0.645 (.011)               | 0.852 (.016)               | 0.579 (.014)               | 0.801 (.020)               | 0.736 (.001)               | 0.914 (.007)               | 0.202 (.011)               | 0.076 (.007)               |
|                          | last dom. wo. ind.    | 0.670 (.005)               | 0.867 (.004)               | 0.609 (.004)               | 0.823 (.011)               | 0.760 (.003)               | 0.915 (.019)               | 0.181 (.003)               | 0.128 (.007)               |

Upon reviewing these results, we acknowledge that the impact of the T2V component appears minimal and statistically insignificant. However, it is worth noting that, aside from the ROC metric, which remains on par, each OOD performance metric shows a slight improvement in mean values when T2V is included. Moreover, we try to be reproducible in using the best configuration our HPO found, which included T2V.

We believe that our final response has addressed all outstanding concerns. We noticed that the mentioned second score increase has not yet been reflected in the review, and with the discussion period ending shortly, we wanted to bring this to your attention. We completely understand if the score was meant to be adjusted only after this final response, and we are grateful for any score you deem appropriate. We simply wanted to ensure that nothing was overlooked as the discussion concludes.

Thank you again for all your feedback, which has helped to improve our work considerably. If additional adjustments or clarifications are needed, we are more than happy to address and incorporate them.

Best regards, The Authors
Dear Reviewer LELq,

We are pleased that this discussion has clarified the misunderstanding and that you recognize how these baselines further strengthen our proposed method.

We have indeed referenced these results in the main paper within Section 4 — Baseline Setup (L260 - L261), stating:

“[...] Methods from the  Wild-Time benchmark are examined separately and detailed in Section A.10.7 of the Appendix. [...]”

However, we agree that this could be extended, and the results are located quite deep in the Appendix. Unfortunately, due to space constraints, we are not able to include the whole results table of Wild-Time into the main part of our work. Nevertheless, (1) we add a table of contents at the beginning of the Appendix and (2) reposition the Wild-Time results so they are more prominently featured. Additionally, (3) we add the best DG method alongside ERM and SWA in Table 1. Furthermore, (4) we add a short discussion of our Wild-Time findings in Section 4 - Quantitative Evaluation, and (5) we explicitly list Wild-Time as a baseline in the abstract.

We believe that the revisions make these important baselines more accessible to readers and further highlight the robustness of our approach.

> I am not sure whether the experiments support this conclusion; it assumes that the cause is the sample complexity of the DG methods (which there are no experiments designed to assess, on my reading) and also that this failure is somehow due to their neural network architecture (also not demonstrated with experiments, this would require conducting a controlled experiment with different architectures in DG models, which is probably far beyond the scope of this paper).

We fully agree that this statement in our response should have included a qualifier like “possibly” or “likely”. Furthermore, we are aware that this is mostly speculation and should only be used as a possible explanation that needs to be validated before stating it as a fact. For this reason, we included a more cautious statement when writing our paper (L736 - L740), where we describe this difference as "likely".

> Are the ERM and SWA methods also neural networks, or are they strictly linear models?

Yes, both the ERM and SWA methods, along with the other methods in the Wild-Time Benchmark, were applied to an MLP. The hyperparameters for these models were optimized following the same process we applied to the other baselines.
The continued commitment by the authors is appreciated. The robustness of the model despite ablation of the domain indices is good to know. The shots experiment I think is also great to show, as it gives a practical sense of how responsive the model could be if used in real world settings.
Dear Reviewer m2Lp,

We thank you again for the time and effort you invested in reviewing our submission, and hope our rebuttal has addressed your questions and concerns regarding the paper. However, if there is anything you still like to see addressed or clarified, we are more than happy to provide additional details until the end of the discussion period.

Best regards, The Authors
Dear Reviewer aHHC,

Thank you for your response and willingness to discuss the paper further with the reviewers. We also greatly appreciate your acknowledgement of our efforts to improve the related work section.

The novelty as mentioned in your response is indeed a subjective topic. However, we believe that our approach of using a second-order SCM within the TabPFN framework, to successfully address temporal distribution shifts in tabular data is indeed very relevant. Although the concept is quite intuitive, the actual implementation involved considerable complexity. Designing the second-order SCM, along with specifying and optimizing all relevant hyperparameters and constructing the necessary sampling spaces, were complex tasks that demanded substantial effort and validation to achieve practical performance. Moreover, the validation of this approach further demonstrates its potential as an effective modeling strategy for future methods outside TabPFN.

Our empirical results highlight that our modifications to TabPFN can lead to substantial improvements in handling real-world and synthetic datasets with temporal distribution shifts. The performance gains of our method, especially in the synthetic case, are unmatched by existing approaches in our setting and are in our opinion a significant step forward. 

Best regards,
The Authors
Dear Reviewer 84bT,

We are glad that our previous clarifications addressed most of your questions, and appreciate that you have raised your score. As promised, we are now providing the results of the two outstanding experiments we mentioned.
> Considerations on ID capabilities when adopting Drift-Resilient TabPFN compared to original TabPFN. Is DSTabPFN guessing a distribution shift when there is none? 

For this, we conducted an analysis on strict synthetic i.i.d. tabular classification datasets. We first instructed our model that all instances in the train and test split belong to the same domain, which aligns with the ground truth. Then, we informed the model that each group of 10 instances belongs to a different domain in increasing order. Therefore, the test instances are pretended to belong to different domains than the training instances. Our preliminary experiments across multiple datasets show that incorrectly reporting domain indices does not significantly impact our model's performance. Below is a table, showing the mean scores across three model initializations for one example dataset:

| Metric | Same Domain (Mean ± 95% CI) | Different Domains (Mean ± 95% CI) |
|--------|-----------------------------|----------------------------------| 
| ROC AUC | 0.9031 (0.0043) | 0.9030 (0.0045) | 
| ACC | 0.8017 (0.0259) | 0.7983 (0.0072) | 
| ECE | 0.0512 (0.0278) | 0.0531 (0.0389) | 
| F1 | 0.8006 (0.0323) | 0.7966 (0.0147) |

These results indicate that our model does not overly rely on domain indices to infer a shift when there is none across the training data. We appreciate your suggestion to conduct this experiment. We will continue to analyze this further and include these findings in the appendix of the camera-ready submission.

> A related analysis that could have been nice to have is to see how many “shots” in the shifted distribution are needed for DSTabPFN to adjust appropriately.

For this analysis, we evaluated the performance of both our model and the baseline TabPFN by fixing the prediction to the last domain of a dataset and gradually increasing the number of domains seen during training. Our results confirm observations made during model development: our method requires significantly less training domains to accurately extrapolate shifts to the distant future, whereas the baseline TabPFN only has acceptable decision boundaries when given data close to the testing domain. Below is the results table for this experiment on the Intersecting Blobs dataset, which was discussed in the qualitative evaluation section of our paper. There, we fixed the predictions to domain $\mathcal{C}^{\text{test}} = \\{9\\}$ and increased the train set from training on domains $\mathcal{C}^{\text{train}} = \\{0, 1\\}$ to $\mathcal{C}^{\text{train}} = \\{0, 1, …, 8\\}$ gradually. The columns in the table represent $\mathcal{C}^{\text{train}}$ that the model was fitted on for the respective scores.

| Metric | $\\{0,1\\}$ | $\\{0,…,2\\}$ | $\\{0,...,3\\}$ | $\\{0,…,4\\}$ | $\\{0,…,5\\}$ | $\\{0,…,6\\}$ | $\\{0,…,7\\}$ | $\\{0,…,8\\}$ |
|--------|-------|-------|-------|-------|-------|-------|-------|-------|
| ROC AUC (TabPFN$_{\mathrm{dist}}$) | 0.7975 | 0.9273 | 0.9994 | 0.9997 | 0.9999 | 1.0000 | 1.0000 | 1.0000 |
| ROC AUC (TabPFN$_{\mathrm{base}}$) | 0.4551 | 0.4855 | 0.3495 | 0.4281 | 0.4581 | 0.7856 | 0.9996 | 1.0000 |
| Acc. (TabPFN$_{\mathrm{dist}}$) | 0.4972 | 0.6556 | 0.9083 | 0.9722 | 0.9917 | 0.9944 | 1.0000 | 0.9917 |
| Acc. (TabPFN$_{\mathrm{base}}$) | 0.5028 | 0.4194 | 0.3028 | 0.2667 | 0.2417 | 0.4833 | 0.9806 | 0.9944 |
| ECE (TabPFN$_{\mathrm{dist}}$) | 0.2994 | 0.2782 | 0.2551 | 0.2649 | 0.1554 | 0.0555 | 0.0218 | 0.0105 |
| ECE (TabPFN$_{\mathrm{base}}$) | 0.2840 | 0.2658 | 0.3404 | 0.4109 | 0.5202 | 0.2539 | 0.1865 | 0.0211 |
| F1 (TabPFN$_{\mathrm{dist}}$) | 0.4972 | 0.6556 | 0.9083 | 0.9722 | 0.9917 | 0.9944 | 1.0000 | 0.9917 |
| F1 (TabPFN$_{\mathrm{base}}$) | 0.5028 | 0.4194 | 0.3028 | 0.2667 | 0.2417 | 0.4833 | 0.9806 | 0.9944 |

Thank you again for suggesting this valuable analysis. We will include the quantitative results for this experiment in the form of line plots, along with decision boundaries, in the appendix of the camera-ready version.
Thanks to the authors for the clarification. 

I indeed did not see these results (they are on page 25 of the paper deep in the appendix, and if they are mentioned in the main text, I perhaps missed it). I was also a bit confused by the author response (it was unclear that these were *already* in the paper, my interpretation was that the authors had *added* them to the paper in response to the discussion). In any case, these are important results and should absolutely be in the main text - these are the most appropriate baselines for the current method. I don't see why they should be buried in the appendix just because they are negative results (and indeed, it makes the proposed method look much stronger). The fact that this reviewer completely missed these results even on a careful reading of the paper indicates that it would probably be helpful to also revise the main text, and perhaps even abstract/intro, to clearly signpost that these baselines are included.

I will further raise my score.

One final note: the authors say in the response above that 

> Our findings indicate that, due to the specific context of small-tabular datasets subject to temporal distribution shifts, neural network approaches struggle to generalize effectively because of the limited amount of training instances

I am not sure whether the experiments support this conclusion; it assumes that the cause is the sample complexity of the DG methods (which there are no experiments designed to assess, on my reading) and also that this failure is somehow due to their neural network architecture (also not demonstrated with experiments, this would require conducting a controlled experiment with different architectures in DG models, which is probably far beyond the scope of this paper). Are the ERM and SWA methods also neural networks, or are they strictly linear models? I would suggest that the authors offer this as a hypothesis, but be clear that this is speculative and not suggest it is somehow "indicated" by their results.
Dear Reviewer LELq,

Thank you very much for your feedback on the related work, and we are happy that you like it. Regarding the baselines, we believe that there might be a misunderstanding that we would like to resolve. We *have evaluated all applicable Wild-Time methods* (ERM, FT, EWC, SI, A-GEM, CORAL-T, GroupDRO-T, IRM-T, Mixup, LISA, SWA) on our Test-Datasets, including performing hyperparameter optimization as described in our initial rebuttal response.

The results of this evaluation are in Section A.10.7 *”Quantitative Analysis: Comparison Against Wild-Time Methods”* of the appendix and we also did a comparison with DRAIN and GI on one dataset in Section A.10.8. of the appendix. 
We would like to understand what you exactly believe is missing? Is there a particular baseline that is not represented in our evaluation, or would you like this section to be inside the main paper? In our initial submission, we decided against this due to the size constraints of the paper and the results being uncompetitive.

We hope that we can clear the misunderstanding and look forward to your response.
Thank the authors for further development of related works. I raised my score.
Thank you to the authors for the detailed follow-up.

The rewritten related work section is a huge improvement and addresses my concerns with that part of the paper. In particular, it properly surveys related work (wrt tabular benchmarking, domain generalization methods, and TabPFN variants) and I think is also quite useful in more properly positioning the work.

I understand that it is difficult to add so many baselines and to add more iterates of the ablation study during the revision window. However, I also feel that these are now the main missing piece of the paper, as they are critical to empirically assessing the proposed method. I will increase my score to 6.
### Missing domain generalization (DG) baselines:
> The author response says that "Previous works, such as TableShift and WildTime, have shown that no other DG method consistently outperforms GBDTs on tabular data." This is not mentioned in the submitted version of the paper to my reading, which is a part of why the paper's comparison to related work seemed to lacking. 

We agree that the comparison to related work and its findings should have been more thoroughly discussed in the initial submission. To address this, we now include a discussion of these studies in the rewritten “Related Work” section.

> Additionally, while I agree that these works provide a strong prior, these other studies are not a substitute for performing these comparisons on the authors' data -- particularly when this is a domain shift method, and there are no domain shift or domain generalization baseline methods in the empirical experiments at all (the authors only compare to vanilla supervised methods like XGBoost and ERM which are not designed for domain shift).

We completely agree that prior work has to be re-evaluated by us and we cannot rely just on their reported findings. As we said in the initial rebuttal, we have rerun all of the baselines in the Wild-Time benchmark on our data to gather results for Table 6 of the appendix. Moreover, we included the two best-performing methods from the Wild-Time benchmark in the main results in Table 1.

Our findings indicate that, due to the specific context of small-tabular datasets subject to temporal distribution shifts, neural network approaches struggle to generalize effectively because of the limited amount of training instances. Standard ERM and SWA performed best among the Wild-Time methods, which is consistent with the findings of previous work. For a more detailed argument, please refer to point 6, "Inclusion of relevant baselines," in our initial rebuttal response.

We look forward to your feedback on these updates and any further suggestions you might have.
Dear Reviewer LELq,

Thank you for your response and for your continued engagement with our work.

### Framing
> It is hard to assess the degree of the authors' success in "rewriting the related work section from the ground up" as the new section does not appear to have been shared. I unfortunately cannot give much consideration to a promise to rewrite this section which is currently very lacking for reasons I outline in my initial review - but the related work section is perhaps not the most significant issue raised in the review (although it is a major one). I have similar reservations about the authors' promised revisions regarding dataset details in the main text.

We appreciate your comments regarding the related work section and dataset details. To address your concerns, we have completely rewritten the related work section and would now like to share these updates with you. We’ve added a global comment for all reviewers in the top, showcasing the new related work section. It now includes discussions on related benchmarks, DG methods, and recent studies on TabPFN. We would greatly value any feedback you could provide to ensure we’ve effectively addressed the issues you previously highlighted.

We did not include the works by Breejen et al. [5] and Rubachev et al. [8], as they were published after our submission. However, if you think it would be helpful to include these references, we are more than willing to reconsider and incorporate them.

> [5] Breejen, Felix den, et al. "Why In-Context Learning Transformers are Tabular Data Classifiers." arXiv preprint arXiv:2405.13396 (2024).

> [8] Rubachev, Ivan, et al. "TabReD: A Benchmark of Tabular Machine Learning in-the-Wild." arXiv preprint arXiv:2406.19380 (2024).

Regarding the revisions to the dataset details, we have incorporated the changes, which were already detailed in our rebuttal. Specifically, in Section 4 - Experiments (line 251), we added the following paragraph:

“[...] While some of these datasets have been analyzed in previous work, there has been no comprehensive benchmark focusing on small tabular datasets undergoing distribution shifts. To address this gap, we carefully selected and generated a diverse range of datasets that exhibit temporal distribution shifts. The datasets were selected from open dataset platforms or previous work in DG. [...]”

Additionally, as mentioned in our rebuttal, all dataset details are thoroughly documented in Section A.11 of the appendix. We hope these revisions meet your expectations, but we would be glad to hear if there’s anything more you would like to see addressed.

### Ablation study and Time2Vec (T2V): 
> Similarly, I do not see updated ablation study results. The authors acknowledge that their submitted result "does not show a significant improvement due to the Time2Vec encoding". Indeed, T2V is worse according to ECE, and only improves AUC by 0.001 (with an error of +/- 0.006) according to Table 2. The other results (accuracy, F1) in this table are on the edge of statistical significance. Again, there are also no error estimates for the no-T2V variant because the authors only perform one iteration. This to me is a major weakness of the ablation study, as this is the purpose of an ablation study (to identify which components of a new method do and do not contribute to its performance). I remain concerned about adding the T2V component to this model when it appears to do little, if nothing at all, and the limited/partial experimental results do not help to assess the extent of this issue.

We fully understand your concerns regarding the missing runs for the no-T2V variant, and agree that these should be included. As promised, we have been running the two outstanding experiments, and while they have not yet completed due to cluster issues, the results should be available by tomorrow morning. We will share these results with you as soon as they are ready.
- [37]  David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex). In M. Meila and T. Zhang, editors, Proceedings of the 38th - International Conference on Machine Learning (ICML’21), volume 139 of Proceedings of Machine Learning Research. PMLR, 2021. URL https://proceedings.mlr.press/v139/krueger21a.html.
- [38]  Cian Eastwood, Alexander Robey, Shashank Singh, Julius von Kügelgen, Hamed Hassani, George J. Pappas, and Bernhard Schölkopf. Probable domain generalization via quantile risk minimization. In Oh et al. [52]. URL https://proceedings.neurips.cc/paper_files/ - paper/2022/file/6f11132f6ecbbcafafdf6decfc98f7be-Paper-Conference.pdf.
- [39]  Andrey Malinin, Neil Band, Alexander Ganshin, German Chesnokov, Yarin Gal, Mark J. F. Gales, Alexey Noskov, Andrey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkov, Vatsal Raina, Vyas Raina, Denis Roginskiy, Mariya Shmatova, Panos Tigar, and Boris - Yangel. Shifts: A dataset of real distributional shift across multiple large-scale tasks. arXiv preprint arXiv:2107.07455, 2021.
- [40]  Andrey Malinin, Andreas Athanasopoulos, Muhamed Barakovic, Meritxell Bach Cuadra, Mark J. F. Gales, Cristina Granziera, Mara Graziani, Nikolay Kartashev, Konstantinos Kyriakopoulos, Po-Jui Lu, Nataliia Molchanova, Antonis Nikitakis, Vatsal Raina, Francesco - La Rosa, Eli Sivena, Vasileios Tsarsitalidis, Efi Tsompopoulou, and Elena Volf. Shifts 2.0: Extending the dataset of real distributional shifts, 2022. URL https://arxiv.org/abs/2206.15407.
- [41]  Jiashuo Liu, Tianyu Wang, Peng Cui, and Hongseok Namkoong. On the need for a language describing distribution shifts: Illustrations on tabular datasets. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, - 2023.
- [42]  Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. In Ranzato et al. [54].
- [43]  Sergey Kolesnikov. Wild-tab: A benchmark for out-of-distribution generalization in tabular regression, 2023. URL https://arxiv.org/abs/2312.01792.
- [44]  Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In Proceedings of the International Conference on Learning Representations (ICLR’21), 2021. URL https: //openreview.net/forum?id=lQdXeXDoWtI. Published online: iclr.cc.
- [45]  Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip S. Yu. Generalizing to unseen domains: A survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering, 35(8):8052–8072, - 2023. doi:10.1109/TKDE.2022.3178128.
- [46]  Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In A. Globerson and R. Silva, editors, Proceedings of The 34th Uncertainty in Artificial - Intelligence Conference (UAI’18). AUAI Press, 2018.
- [47]  Junwei Ma, Valentin Thomas, Guangwei Yu, and Anthony Caterini. In-context data distillation with tabpfn, 2024. URL https://arxiv.org/abs/2402.06971.
- [48]  Benjamin Feuer, Robin Tibor Schirrmeister, Valeriia Cherepanova, Chinmay Hegde, Frank Hutter, Micah Goldblum, Niv Cohen, and Colin White. Tunetables: Context optimization for scalable prior-data fitted networks. arXiv preprint arXiv:2402.11137, 2024.
- [49]  Samuel Dooley, Gurnoor Singh Khurana, Chirag Mohapatra, Siddartha V Naidu, and Colin White. Forecastpfn: Synthetically-trained zero-shot forecasting. In Advances in Neural Information Processing Systems, 2023.
- [...]
**Recent Temporal DG methods.** Recent specialized temporal DG methods include: (1) DRAIN [12], which employs a Bayesian framework alongside a recurrent neural network for predicting the dynamics of the model parameters across temporal domains. (2) GI [13], which explicitly incorporates the temporal domain as a feature, using a specialized Gradient Interpolation loss function, a time-sensitive activation, and enhanced domain reasoning via Time2Vec preprocessing [22]. However, both DRAIN and GI are limited in their ability to extrapolate beyond the near future, leading to their exclusion from our main evaluation. Notably, even on the Rotated Two Moons Dataset - where DRAIN and GI have demonstrated their capabilities - our approach Drift-Resilient TabPFN, outperforms both. See Appendix A.10.8 for details.

**Recent TabPFN Studies.** To overcome the limitations of TabPFN in handling large samples and feature sets, typically found in DG benchmarks, several improvements have been proposed. Ma et al. [47] introduced a data distillation approach, where a distilled dataset serves as the model’s context, optimized to maximize the training-data likelihood. Similarly, TuneTables [48], uses prompt-tuning to compress large datasets into a smaller context. Either of these methods could potentially be combined with Drift-Resilient TabPFN, as our modifications focus primarily on the pre-training phase, which remains unchanged in their approaches.

Another TabPFN variation, ForecastPFN [49], also introduces time dependence, but it does not consider any features. It only models simple time series data. Unlike our approach, which builds on TabPFN’s SCM architecture for pre-training to handle a large set of features, ForecastPFN models synthetic time series using a single handcrafted function with sampled hyperparameters, simplifying the architecture while trading off diversity of the synthetic datasets.

### References:
- [...]
- [11]  Huaxiu Yao, Caroline Choi, Bochuan Cao, Yoonho Lee, Pang Wei Koh, and Chelsea Finn. Wild-time: A benchmark of in-the-wild distribution shift over time. In Oh et al. [52]. URL https://openreview.net/forum?id=F9ENmZABB0.
- [12]  Guangji Bai, Chen Ling, and Liang Zhao. Temporal domain generalization with drift-aware dynamic neural networks. In Proceedings of the International Conference on Learning Representations (ICLR’23) icl [53]. URL https://openreview.net/forum?- id=sWOsRj4nT1n. Published online: iclr.cc.
- [13]  Anshul Nasery, Soumyadeep Thakur, Vihari Piratla, Abir De, and Sunita Sarawagi. Training for the future: A simple gradient interpolation loss to generalize along time. In Ranzato et al. [54]. URL https://openreview.net/forum?id=U7SBcmRf65.
- [14]  Josh Gardner, Zoran Popovic, and Ludwig Schmidt. Benchmarking distribution shift in tabular data with tableshift. Advances in Neural Information Processing Systems, 36, 2024.
- [...]
- [22]  Seyed Mehran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jaspreet Sahota, Sanjay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, and Marcus Brubaker. Time2vec: Learning a vector representation of time, 2020. URL https://openreview.net/forum? - id=rklklCVYvB.
- [...]
- [29]  Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain generalization using meta-regularization. In Bengio et al. [55].
- [30]  Saeid Motiian, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto. Unified deep supervised domain adaptation and generalization. In Proceedings of the IEEE international conference on computer vision, pages 5715–5725, 2017.
- [31]  Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk mini- mization, 2020.
- [32]  Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In Proceedings of the International Conference on Learning - Representations (ICLR’20), 2020. Published online: iclr.cc.
- [33]  Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In Proceedings of the International Conference on Learning Representations (ICLR’18), 2018. Published online: iclr.cc.
- [34]  Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving out-of-distribution robustness via selective augmentation, 2022.
- [35]  Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Gang Hua and Hervé Jégou, editors, Computer Vision – ECCV 2016 Workshops, pages 443–450, Cham, 2016. Springer International Publishing. doi:10.1007/- 978-3-319-49409-8_35.
- [36]  Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(1):2096–2030, jan 2016. - ISSN 1532- 4435.
### Related Work
While DG has drawn increasing attention in the research community [28–38], its temporal variant remains under-explored. In this section, we review existing DG benchmarks on tabular data, DG methods, the temporal DG benchmark Wild-Time [11], specialized temporal DG methods, as well as relevant studies of TabPFN.

**Tabular Distribution Shift Benchmarks.** Several benchmarks have been introduced to assess methods for tabular data under distribution shifts. (1) **Shifts** and **Shifts 2.0** [39, 40] are uncertainty focused benchmarks. **Shifts 2.0** includes five tasks, two of which involve tabular data subject to temporal and spatio-temporal shifts. (2) **WhyShift** [41], focusing on spatio-temporal shifts, offers five real-world tabular datasets, including the ACS dataset [42], which we also use in a subsampled form. Evaluating 22 methods like Gradient Boosted Decision Trees (GBDTs), MLPs, and robustness techniques, WhyShift finds that robustness methods do not consistently enhance out-of-distribution (OOD) performance. They also find that while GBDTs perform better, the gap between in-distribution (ID) and OOD performance persists, likely due to GBDTs being better fitted to the ID distribution. (3) **TableShift** [14] includes 15 tabular binary classification tasks, with 10 relevant to DG. Their work evaluates 19 model types, including several DG methods, finding that methods tailored for distribution shifts do not consistently outperform GBDTs. While generalization gaps can be slightly reduced, each robustness method comes at the cost of ID performance. (4) **Wild-Tab** [43] focuses on domain generalization within tabular regression using three large datasets. Their study compares 10 generalization techniques against standard Empirical Risk Minimization (ERM) applied to MLPs. Similar to previous work [44], they find that ERM was not consistently outperformed by specialized DG methods, and notably, no advantage of GBDTs over ERM on MLPs was observed in their datasets.

Unlike these benchmarks, which focus on large-scale datasets, our work addresses the overlooked challenges of small-scale temporal distribution shift datasets. However, their insights on generalization and robustness methods closely align with our findings, providing valuable context for our approach.

**DG Methods.** Several techniques have been proposed to improve robustness to distribution shifts in DG [45]. Key approaches include domain-invariant learning methods, such as **Deep CORAL** [35], **IRM** [31], and **DANN** [36], which aim to learn representations that generalize across domains. Data augmentation strategies, like **Mixup** [33] and **LISA** [34], contribute to generalization by generating synthetic data variations. Additionally, robust optimization techniques, including **VRex** [37], **GroupDRO** [32], **EQRM** [38], and **SWA** [46], aim to improve performance under distributional shifts by optimizing for worst-case scenarios or incorporating model uncertainty.

In relation to our work, data augmentation strategies are conceptually similar, as we also teach the model DG rather than adding invariances to the architecture. In contrast to augmentation strategies, though, we learn to generalize using completely artificial data on the meta level rather than through manipulation of the target dataset. The other methods focus on designing models specifically for DG, while our approach completely relies on the model to learn to handle distribution shifts.

**Wild-Time Benchmark.** Wild-Time [11] is a benchmark of five datasets designed to study the real-world effects of temporal distribution shifts - an area largely overlooked by previous benchmarks. While Wild-Time primarily uses non-tabular data, it evaluates a wide array of techniques on its tabular dataset, including classical supervised learning (ERM), fine-tuning, and several previously mentioned general DG methods adapted to temporal distribution shifts. Despite the diversity of methods evaluated, Wild-Time reveals a significant performance gap between ID and OOD data, with none of the 13 tested methods consistently outperforming the standard ERM approach.

In our evaluation, we employ their evaluation strategy Eval-Fix and also benchmark our approach against the methods they considered. A comprehensive overview of these methods is provided in Appendix A.8.
Dear Reviewers,

We agree with Reviewer LELq's observation that our original "Related Work" section was insufficiently detailed, primarily due to space limitations. With the additional page provided for the rebuttal, we have significantly expanded and reworked this section. The revised version now offers a more comprehensive overview of related benchmarks, general DG methods, and recent studies on TabPFN. We also detail relevant findings and clarify how they relate to our work. Given the importance of this section, we are sharing the updated “Related Work” section with all of you in the comment below.

We believe these revisions address the primary concerns regarding this section, and we would appreciate any further feedback you may have to ensure clarity and completeness.
Thank very much the authors for clarifying the terminology. Indeed, hypernetworks are commonly referred as another methodology.

I still feel the improvement wrt TabPFN is bit incremental. But this is very subjective. At any rate, I don't feel a strong novelty in the second order SCM used here, as how SCM is used in Domain Generalization / Invariance Learning literatures.

I am happy to discuss with other reviewers later on it and raise the score if needed.
Thank you to the authors for the detailed response. I have reviewed both the authors' overall response, and the authors' individualized response to my review. The authors seem to have addressed some of my concerns around title and framing, along with the resources dedicated to baselines (their findings wrt saturation are in line with other tabular studies).

* **Framing**: It is hard to assess the degree of the authors' success in "rewriting the related work section from the ground up" as the new section does not appear to have been shared. I unfortunately cannot give much consideration to a promise to rewrite this section which is currently very lacking for reasons I outline in my initial review - but the related work section is perhaps not the most significant issue raised in the review (although it is a major one). I have similar reservations about the authors' promised revisions regarding dataset details in the main text.

* **Ablation study aind Time2Vec (T2V)**: Similarly, I do not see updated ablation study results. The authors acknowledge that their submitted result "does not show a significant improvement due to the Time2Vec encoding". Indeed, T2V is worse according to ECE, and only improves AUC by 0.001 (with an error of +/- 0.006) according to Table 2. The other results (accuracy, F1) in this table are on the edge of statistical significance. Again, there are also no error estimates for the no-T2V variant because the authors only perform one iteration. This to me is a major weakness of the ablation study, as this is the purpose of an ablation study (to identify which components of a new method do and do not contribute to its performance). I remain concerned about adding the T2V component to this model when it appears to do little, if nothing at all, and the limited/partial experimental results do not help to assess the extent of this issue.

* **Missing domain generalization (DG) baselines**: The author response says that "Previous works, such as TableShift and WildTime, have shown that no other DG method consistently outperforms GBDTs on tabular data." This is not mentioned in the submitted version of the paper to my reading, which is a part of why the paper's comparison to related work seemed to lacking. Additionally, while I agree that these works provide a strong prior, these other studies are not a substitute for performing these comparisons on the authors' data -- particularly when this is a domain shift method, and there are no domain shift or domain generalization baseline methods in the empirical experiments at all (the authors only compare to vanilla supervised methods like XGBoost and ERM which are not designed for domain shift).

I remain open to revising my score upward, despite these outstanding issues. I would like to discuss with the reviewers during the dialogue window.
Thank you for the detailed response. I believe most of my initial questions have been resolved.
I don't want to create more work for the authors in a short period of time, the work as is seemed impactful to me and the future directions seem interesting. 
After deliberation the work seems clearer to me (and I feel that some questions about related works have also been resolved in the other rebuttals), so I would be willing to raise the score.
> Why do sparse shifts allow for causal reasoning (Line 195)?

For details, we refer to the foundational work of Perry et al. [b], which addresses the challenge of causal discovery from observed data. In traditional settings with i.i.d. data from non-shifted SCMs, the recovery of causal graphs is limited to the Markov equivalence class due to the symmetrical factorization of joint distributions.

Perry et al. demonstrate that data generated from SCMs experiencing sparse shifts can distinctly break this symmetry, enabling the identification of causal structures beyond the Markov equivalence class. They provably show that sparse shifts serve as a learning signal for inferring causal relationships, whereas unrestricted shifts cannot be traced in the data.
These results are important for our approach. They indicate that our transformer can learn to extract and utilize causal relationships from the sparse shifts in our prior. During pre-training, the model encounters these sparse shifts, allowing it to infer causal relationships. When presented with sparsely shifted data during training, the transformer can apply this learned causal reasoning during inference. This theoretical foundation underpins the effectiveness of our approach and supports the validity of our method as outlined in our paper.

> How are the edge weights decided?

The edge weights of a sampled functional representation graph are initially randomized using a well-known weight initialization technique (e.g., Xavier or Kaiming). We then determine randomly whether each causal relationship will shift over time. For the edges that are shifted, we randomly decide whether the shift will be multiplicative or additive. The parameters for scaling or shifting the weights are then sampled from our "hypernetwork", a secondary SCM, using the corresponding time index $t$ as input. This results in the functional representation graph for time $t$, from which instances can be sampled by inputting random noise. We hope this clarifies how the shifts are applied to the functional representation $\tilde{\mathcal{G}}$. For additional details, please refer to Section 3.2 and Algorithm 1 in A.7 of the appendix. If you have any suggestions for improving clarity, please feel free to share them with us. We are willing to revise this section to facilitate understanding while maintaining the right balance between intuition and mathematical rigor.

> Why can SCM with NN-based causal mechanisms and nonlinear activations extrapolate values out of data distribution (Line 208)? Is this due to a causal mechanism or NN?

This ability to extrapolate values out of the observed data distribution stems from our use of a prior that assumes data can be explained by a simple SCM with NN-based functions, rather than employing a specific trained neural network. 
In our framework, the first-order SCM models the data generation process, while the second-order SCM (which we initially termed a "hypernetwork") captures trends in how the first-order SCM's parameters change over time. This second-order SCM doesn't directly predict data points, but rather models the evolution of the data-generating process itself.
The extrapolation occurs because the second-order SCM picks up on trends in weight changes that could explain the observed changes in the first-order SCM's target mapping. These trends are then extrapolated according to our prior, which favors simple trends (i.e., those that can be described by NN functions with few parameters).
This approach allows for extrapolation that is both flexible (due to the use of neural network-based functions) and constrained (by the simplicity prior and the causal structure imposed by the SCMs). It's this combination that enables meaningful extrapolation beyond the observed data distribution, while maintaining the underlying causal relationships and favoring simpler explanations for distributional changes.

We hope we have addressed all your questions and concerns. If we have addressed your concerns, we would very much appreciate it if you could consider increasing your score. Thank you again for your valuable feedback.

### References:
- [a] Judea Pearl. Causality. Cambridge University Press, 2 edition, 2009.
- [b] Ronan Perry, Julius Von Kügelgen, and Bernhard Schölkopf. Causal discovery in heterogeneous environments under the sparse mechanism shift hypothesis. In Proceedings of the 36th International Conference on Advances in Neural Information Processing Systems (NeurIPS’22), 2022.
> What if there are some outliers in the inference phase despite the underlying distribution will continue longer term?

As for the inference phase, an outlier in the predictions would only affect the prediction of that specific outlier, as each prediction is made independently. If we consider a scenario where the outlier is later added to the training set with its ground truth, our model could indeed be influenced by it. However, it is important to note that outliers can also occur during pre-training, so we expect our model to have some capacity to handle them. This is a great suggestion that has sparked discussions among us as coauthors, and it would certainly be interesting to investigate this further!

> What is the rationale for the compute budget imposed in line 262? 

The rationale behind this compute budget is that it is sufficient for the baselines to reach saturation. To demonstrate this, we conducted additional runs of the GBDT methods with a computational budget three times higher than the original, increasing it from 20 minutes to 1 hour. The results in Table 1 of the PDF in the global response to all reviewers show that the OOD performance remains approximately the same as before, indicating that the baselines do not benefit from additional computation time beyond this budget.

> It would be great to understand how the approach scales with the amount of samples from the SCM.

First, both the baseline TabPFN and Drift-Resilient TabPFN were pre-trained on the same number of datasets generated by their respective priors. However, due to the increased complexity of temporally shifted datasets, our model improves performance more slowly than the base TabPFN. During model development, we observed that longer pre-training phases are required to accurately capture the dynamics in our validation datasets, likely due to (1) the difficulty of the task and (2) the need to cover the extensive sample space of our prior fairly.

The minor drop in ID performance is primarily due to our focus on OOD generalization during pre-training, where we only train on the ID task for a small percentage of the generated datasets. This is a tradeoff; we could improve ID performance by dedicating more training to it, at the expense of some OOD generalization.

Regarding the overfit regime, there are two points: First, we can’t overfit during pre-training since we only train on synthetic data and the data used for inferenceis never seen. Second, the longer we train, the more our model adapts to the dynamics generated by our prior. In this sense, one could say we overfit to these dynamics. 

> Should users rely on the model to distinguish between one shift to another or only turn to DSTabPFN when a distribution shift is noticed.

As of now, we recommend applying Drift-Resilient TabPFN when a temporal distribution shift is expected in the data.
However, the question raises an interesting point about the model's applicability. While Drift-Resilient TabPFN is primarily designed for scenarios where temporal shifts are expected, it can potentially offer benefits even in cases where shifts are not explicitly anticipated. As described in Section 5 of our paper, we envision a more relaxed version of our approach that could extend the base TabPFN to employ a prior modeling sparse temporal shifts without requiring explicit domain indices. This extension could enhance robustness in standard classification tasks where temporal shifts may be present but not explicitly identified. Additionally, Drift-Resilient TabPFN could be integrated into stacking ensembles to leverage its shift-awareness alongside other models, and time series cross-validation could be employed to provide more robust performance estimates in temporal settings.

> A related analysis that could have been nice to have is to see how many “shots” in the shifted distribution are needed for DSTabPFN to adjust appropriately.

We appreciate this insightful question. During model development, we observed that adding instances from just one additional domain to the training set significantly improved predictions. In addition, our method demonstrated the ability to already extrapolate shifts far into the future with only a few domains included in the training dataset.

To provide experimental insights on this, we have set up an experiment where we fix the target domain (e.g., the last domain in the dataset) and incrementally add one domain at a time to the training set, starting from two training domains. We will then plot our model's performance and calibration for the last domain based on the current training set. We share the results of this experiment as they are available.

We hope to have addressed your questions adequately and welcome any further discussions you might have. If we have addressed your concerns, we would very much appreciate it if you could consider raising your score. Thank you again for your thoughtful review, which has sparked ideas for follow-up analyses.
### 6. Inclusion of relevant baselines
> Lack of relevant baselines: I was quite surprised to see that the authors do not include any domain generalization or robustness methods in their study. [...]

We had already included the strongest among these methods in our main results (Table 6) and fully in the appendix (Section A.10.7 and Section A.10.8), referenced from our main text. We will more prominently feature these methods in our camera ready. Due to their weak performance (as previously found by the authors of the TableShift and WildTime benchmark) we did not discuss these results in great detail. As you mentioned, Wild-Time provides open-source implementations, and we have spent considerable time adapting these implementations to work within our evaluation framework. We evaluated all Wild-Time methods applicable to tabular data and included the best-performing methods (classical ERM and SWA) in the results table of the main paper.

However, as noted previously, due to the small size of tabular datasets and the resulting limited training instances, these methods do not perform nearly as well as GBDTs or TabPFN. This finding is consistent with previous results from Wild-Time and TableShift, which indicate that none of the DG methods consistently outperform GBDTs. We chose to focus on Wild-Time rather than TableShift because, while the evaluated methods largely overlap, Wild-Time specifically focuses on temporal DG, aligning more closely with the aims of our study.

### Minor details:
We appreciate your feedback on providing more details about the datasets in the main text. We have now included the following paragraph in Section 4 - Experiments in line 251 to address this:

“[...] While some of these datasets have been analyzed in previous work, there has been no comprehensive benchmark focusing on small tabular datasets undergoing distribution shifts. To address this gap, we carefully selected or generated a diverse range of datasets that exhibit temporal distribution shifts. The non-generated datasets were selected from open dataset platforms or previous work in DG. [...]”

Additionally, we would like to point out that Section A.11 of the appendix provides an in-depth discussion of each dataset. This includes the nature of each dataset, the types of shifts they contain, the subsampling performed, as well as the approximation of domain indices.

We found CatBoost's strong ID performance interesting as well. However, due to our focus on out-of-distribution performance, we did not highlight it further in our evaluation.

Thank you for pointing out the typos and minor mistakes; we have gladly corrected them.

Overall, we believe your in-depth review and the corresponding changes have significantly improved our paper, and we would be grateful if you could raise our score. Thank you again for your valuable feedback!
### 4. Expanded Ablation Study

> Ablation study: The ablation study should be expanded, and discussed in the main text. In particular: (1) the current results show no benefit from the Time2Vec addition [..](2) the authors seem to have only conducted a single run, of the ablated model [..] (3) please discuss the results in the main text [..]

You are right that our ablation does not show a significant improvement due to the Time2Vec encoding. (1) We chose to include Time2Vec based on a large HPO search, as it performed best there, even though the improvement is not significant. (2) We acknowledge the limitations of our initial single-run ablation study. To ensure statistical significance, we are conducting two additional runs, scheduled for completion between August 11-12, 2024. (3) We will share these results in the rebuttal once available and will include them in the main paper. Additionally, we will add the following sentence in L307 of the Experiment section, summarizing the main insights from our ablation study:

“[...] The ablation reveals that while Time2Vec provides slight improvements, the substantial performance gains are to be attributed to the prior construction used during the model's pre-training phase. [...]”

Beyond the Time2Vec ablation, we are open to suggestions for other parts to ablate. The time-dependent data-generating SCM and the hypernet for weight shifts were primarily tuned through hyperparameter optimization on validation datasets. Given the involvement of about 50 hyperparameters and the intensive computational resources required for pre-training, it is challenging to find a meaningful ablation that would provide valuable insights into our model's performance.

### 5. Resources for Baselines and Drift-Resilient TabPFN

> Resources for baselines: The authors restrict all baselines to 1200 seconds on 8CPU/1GPU, but the proposed method is trained for 30 epochs on 8 GPUs with a random search over 300 configurations. This setup seems to starve the baselines of resources. Please either provide evidence that the baselines have in fact saturated (i.e. they would not be improved from further tuning) or allocate similar resources to the baselines as to the main model -- [...] 

We would like to clarify the resource allocation for Drift-Resilient TabPFN. As mentioned in lines 277-281, the pre-training of our model, which involved 30 epochs on 8 GPUs and preprocessing optimization over 300 configurations, was only done once in advance. This pre-training was conducted solely on synthetic data generated by our prior, and the preprocessing optimization was performed on our validation datasets. All these steps can be considered as part of the algorithm development process and have to be done only once. The same model was applied to each test dataset and can be used for new, unseen datasets.

The actual training of each real test dataset as well as the inference was done on 8 CPUs and 1 GPU, identical to the baselines. For training and inference on the test datasets, our model was on average 110 times faster compared to the baselines. 

To further address the issue of baseline saturation, we trained the three best-performing baselines (CatBoost, XGBoost, and LightGBM) for 3600 seconds and observed nearly identical out-of-distribution performance. The table for these extended baseline runs can be found in Table 1 included in the PDF of the global response to all reviewers.

Regarding neural network approaches: Previous works, such as TableShift and WildTime, have shown that no other DG method consistently outperforms GBDTs on tabular data.
> If I understand correctly, the core part of the method lies in training a hypernetworks to update SCM. All the rest follows TabPFN?

Not quite. As described in Section 3 of our work, while we do employ a "hypernetwork" to update each data generating SCM, this approach is not merely a simple extension of the TabPFN framework. Instead, it introduces substantial changes in three key areas:
1. Temporal Dependence: Unlike TabPFN, which generates each instance within a tabular dataset from the same randomly-sampled SCM during pre-training, our model introduces temporal dependencies within the SCM. We dynamically shift causal relationships as instances are generated, simulating real-world scenarios where underlying models evolve over time.
2. Employing a "Hypernetwork": Our "hypernetwork", which is itself a secondary SCM, samples the weight shift for each shifted causal relationship given a certain time index as input. This leads to the generation of correlated weight shifts that adhere to underlying causal principles.
3. Temporal Encoding: In processing datasets in the transformer, either generated during pre-training or provided during training and inference, we use Time2Vec encoding for the temporal domain indices. This learned representation attempts to effectively capture temporal aspects of data, such as seasonality, enabling the transformer to better utilize time-indexed information.

To address potential confusion arising from our terminology, it’s important to clarify that the "hypernetwork" we refer to is not the typical learned "hypernetwork" commonly used in machine learning. Instead, it is randomly generated and used solely for sampling. To further enhance clarity, we propose renaming it to “$2^{\text{nd}}\text{-order SCM}$” in our work. Additionally, we will add the following clarification after the end of line 202 in Section 3.2:

“[...] Note that although we perform a forward pass, there is no backward pass associated with it. Each $2^{\text{nd}}\text{-order SCM}$ is randomly generated and used solely for sampling the weight shifts. [...]”

> How good has the hypernetworks been trained? Since there is many synthetic datasets used, it is great to study cases where hypernetworks updated the SCM correctly

We apologize for any confusion caused by our use of the term "hypernetwork". To clarify: Our "hypernetwork" is not a traditional trained neural network, but rather a secondary Structural Causal Model (SCM) used for sampling temporal shifts. This secondary SCM is randomly generated, not trained, and serves to produce correlated weight shifts that adhere to underlying causal principles. This modified prior generates datasets with temporal distribution shifts that closely resemble real-world scenarios. During model development, we conducted extensive analyses on the datasets generated from our prior, and observed the three main types of shifts as well as their combination over time (see Figure 4). To calibrate the strengths and types of these shifts, we employed a random search of hyperparameters on our validation datasets.

> It is vague how hypernetworks is trained and processed. It is suggested to provide details on this

As outlined in the previous response, there seems to be confusion when we refer to our secondary SCM, which is used solely for sampling, as the "hypernet". We are glad that this was caught in the review and have updated this term, see above.

> How much data is required to train hypernetworks?

Since our "hypernetworks" are randomly sampled SCMs, there is no training involved. However, modeling the sample space of SCMs requires careful consideration. For this, we followed the insights of the TabPFN paper as well as empirical evidence by performing a random search of hyperparameters on our validation datasets.

> How is the hypernetworks training related to number of features, number of classes, etc. since TabPFN has limitations on the dataset requirement.

Thank you for this important question. To clarify: Our "hypernetwork" is not trained in the traditional sense, see our comments above. The limitations of TabPFN regarding the number of features and classes remain the same for our model, since the hypernet is only involved in the data generation process. The actual transformer training is therefore nearly as efficient as TabPFNs base implementation. Ongoing improvements made to the TabPFN architecture and in-context learning will translate to our line of work as well.

Thank you again for your feedback. We hope our clarifications and the forthcoming revisions will address your concerns adequately. If this is indeed the case, we would very much appreciate it if you considered raising your score.