Understanding the Limits of Deep Tabular Methods with Temporal Shift
Authors: Haorun Cai
Han-Jia Ye
Venue: ICML 2025 poster (2025)
Keywords: Machine learning on tabular data
Temporal shift
Deep tabular learning
Primary area: deep_learning

Abstract: Deep tabular models have demonstrated remarkable success on i.i.d. data, excelling in a variety of structured data tasks. 
However, their performance often deteriorates under temporal distribution shifts, where trends and periodic patterns are present in the evolving data distribution over time.
In this paper, we explore the underlying reasons for this failure in capturing temporal dependencies. 
We begin by investigating the training protocol, revealing a key issue in how the data is split for model training and validation.
While existing approaches typically use temporal ordering for splitting, we show that even a random split significantly improves model performance. 
By accounting for reducing training lag and validation bias to achieve better generalization ability, our proposed splitting protocol offers substantial improvements across a variety of methods.
Furthermore, we analyses how temporal data affects deep tabular representations, uncovering that these models often fail to capture crucial periodic and trend information. 
To address this gap, we introduce a plug-and-play temporal embedding based on Fourier series expansion to learn and incorporate temporal patterns, offering an adaptive approach to handle temporal shifts.
Our experiments demonstrate that this temporal embedding, combined with the improved splitting strategy, provides a more effective and robust framework for learning from temporal tabular data.

Understanding the Limits of Deep Tabular Methods with Temporal Shift
Hao-Run Cai
Han-Jia Ye!
Abstract
Qu Data
Deep tabular models have demonstrated remark- able success i..d. data, excelling in a variety of structured data tasks. However, their perfor- mance often deteriorates under temporal distri- bution shifts, where trends and periodic patterns are present in the evolving data distribution over time. In this paper, we explore the underlying rea- sons for this failure in capturing temporal depen- dencies. We begin by investigating the training protocol, revealing key issue in how model se- lection performs. While existing approaches use temporal ordering for splitting validation set, we show that even random split can significantly improve model performance. By minimizing the time lag between training data and test time, while reducing the bias in validation, our proposed train- ing protocol significantly improves generalization across various methods. Furthermore, we analyze how temporal data affects deep tabular represen- tations, uncovering that these models often fail to capture crucial periodic and trend information. To address this gap, we introduce plug-and-play temporal embedding method based on Fourier se- ries expansion to learn and incorporate temporal patterns, offering an adaptive approach to han- dle temporal shifts. Our experiments demonstrate that this temporal embedding, combined with the improved training protocol, provides a more ef- fective and robust framework for learning from temporal tabular data.
Time
Texin
fn
On Tempical Shift Data
Time
Figure 1. Illustration of the challenges posed by temporal shifts. The change data distribution over time is represented by dots on a line, with the dashed line depicting the underlying data distribution at different time slices. The shaded box indicates the mapping f Iearned from the training data Taaan while the training data is typically treated as i..dd on Airain and Viaain in classical training processes. On i.i.d. data, the model directly apply the learned mapping f to make accurate predictions on test data, but it fails to generalize effectively when temporal shifts occur.
It consists of instances (rows) and features (columns), where the label can either be categorical or continuous, correspond- ing to classification and regression tasks, respectively. The goal to learn a mapping strategy from features to labels that can be directly applied independent and identically distributed (i.i.d)) test data for accurate predictions (Bishop, 2007; Mohri et al., 2012). While tree-based methods remain powerful (Breiman, 2001; Chen & Guestrin, 2016; Ke et al., 2017: Prokhorenkova et al., 2018), recent advancements in deep learning methods have demonstrated promising results (Klambauer et al., 2017; Arik & Pfister, 2021; Wang et al., 2021b; Gorishniy al., 2021; 2024; 2025; Hollmann et al., 2023; 2025; Ye et al., 2023; 2025; Holzmüller al., 2024; Jiang al., 2024; 2025; Liu & Ye, 2025; Liu et al., 2025).
1. Introduction
Most machine learning approaches are built on the assump- tion of i.i.d. data. However, in open environments (Zhou, 2022), distribution shifts (Gardner et al., 2023; Tschalzev et al., 2024) frequently occur between the training and test- ing data, which can manifest shifts in the feature space, label space, the mapping between them. Moreover, in practical applications, data often exhibits temporal distribu- tion shifts (Rubachev et al., 2025), particularly common type of distribution shift that introduces additional chal- lenges: instead of just shifts between training and test sets, temporal shifts can occur within the training set or the test
Tabular data is one of the most prevalent data formats in wide range of real-world applications, such as e-commerce (Nederstigt et al., 2014) and healthcare (Hassan et al., 2020).
'School of Artificial Intelligence, Nanjing University, China "National Key Laboratory for Novel Software Technology, Nanjing University, China. Correspondence Han-Jia Ye <ychj@ laai@almmajjuudu..crm
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Understanding the Limits of Deep Tabular Methods with Temporal Shift
Temporal Split in TabReD
Random Split
547%
5.10%
0.05
4A18
Deresntaes
0.04
1120%
3.30%
3.24%
N.209
3.11/4
0.03
2
+3.00%
3.77
147%
0.02-
+3.4%%
SNN
4.22%
128
MLP
MUPPLA
XC
Cattoooi
LightGBM
DCN-2
Thbo
ModemaCCh
TabM
-0.01
MIP
MLP-PLA
XCRoset
Cathooo
Ligh
SNN
DONv2
FT-7
Tbba
ModemNNCA
TabM
MLP Architacture
Tree-Based
Tolonn-Based
Retriesal-Base
Ensemble-Based
Imprrvveenn
Figure 2. Performance comparison between temporal split in Rubachev et al. (2025) and random split on TabReD benchmark,
where only the data splitting strategy before Twain is changed. The percentage change represents the robust average of performance difference compared the MI. with original temporal split. positive percentage change indicates that the method outperforms the MI.P with temporal split. Left: We reproduce the experiment from Rubachev et al. (2025) and ensure a fair comparison by removing additional numerical feature encodings, as explained in Appendix A. In this setting, the performance of retrieval- based methods significantly declines, falling behind tree-based methods and MLP-PLR, while TabM achieves the best performance. Right: Performance improvements observed under the random split. Retrieval-baa -based methods show the most substantial gains, and model rankings align more closely with conventional expectations. Detailed results are provided in Appendix C.I.
set itself. For example, in house price prediction (Matveev & Sidorova, 2017), the task is to predict future house prices using historical transaction data, which includes features such as location, neighborhood conditions, and economic factors. However, in actual practice, the trend of housing policies or the periodic fluctuations of public sentiment can also play erucial roles in influencing house prices.
validation during the training stage (Blanchard et al., 2021) makes model selection more impactful, as deep learning models are optimized epoch-wise. TabReD employs a tem- poral splitting strategy to match the test scenario, utilizing earlier data for training and later data prior to T Tirain as the validation set for model selection. Surprisingly, we discov- ered that by merely altering the splitting strategy the same training data, even when randomly splitting the train- ing and validation sets and ignoring temporal order, the model outperformed the temporal split, resulting in signif- icantly enhanced performance, as illustrated in Figure 2. Moreover, the performance rankings of the models became more consistent with established results on i.i.d. data, with retrieval- based methods showing more distinct improve- ments. This unexpected finding underscores the urgent need to investigate effective data splitting strategies to accurately assess model performance and reveal its true capabilities in the presence of temporal shifts. Despite the performance im- provement, the random split exhibits pronounced instability in temporal shift context, which also requires attention.
These temporal shifts lead to the failure of the model when the training process assumes the data to be i.idd The chal- lenge of training on temporal shift data is illustrated in Figure 1. Formally, we aim to perform model training and validation using data prior to Tirain and subsequently use the trained model make predictions on data after Tirain- Once the temporal shift occurs, it leads to discrepancies between the mapping learned during training and the one required for deployment, making the model ineffective.
Since temporal shifts are common and present significant challenges, we have turned our attention on whether tab- ular data models that perform well in classic i.i.d. scenar- ios can effectively manage temporal shifts. In recent stud- ies, Rubachev et al. (2025) introduced the TabReD bench- mark. By comparing the performance differences of various methods on the original temporal shift dataset and the i.i.d. dataset constructed through shuffling, their observations re- vealed that while models with MLP architectures exhibit relative robustness during temporal shifts, retrieval- beasee methods, which are highly competitive in current bench- marks (Ye et al., 2024), suffer a marked performance degra- dation in temporal shift scenarios. This observation demon- strates that the occurrence of temporal shifts can introduce significant biases in the evaluation of model performance, further emphasizing the importance of understanding and managing these temporal shifts.
We begin by analyzing the factors contributing to the in- effectiveness of temporal splitting in this context. While temporal- based splitting is commonly employed in fore- casting tasks to maintain causal relationships (Bergmeir & Benítez, 2012), the most intuitive difference when adopt- ing a temporal split in tabular learning lies in the time lag between the training and test sets, and the bias in valida- tion, since the data closest to the test time are not used for training, and the shift degree of test set relative to validation set is more significant in temporal splits. By investigating the impact of these two factors, we find that minimizing the training lag concentrates the model's performance closer to the test time, while reducing validation bias by aligning the shift degree makes the model better generalize on test data. Building on these insights, we propose training pro-
In temporal shift tasks, the absence of accurate test data
2

---

Understanding the Limits of Deep Tabular Methods with Temporal Shift
tocol along with an improved temporal split that leverages these advantages, resulting in comparable performance to random splitting while providing better stability.
et al., 2025) are retrieval-based methods that make predic- tions by retrieving neighbors in the representation space, TabM (Gorishniy et al., 2025) provides an ensemble-based method on MLP, while other methods like SNN (Klambauer et al., 2017), DCNv2 (Wang et al., 2021 and MLP-PI. (Gorishniy et al., 2022) focus on improving the MLP archi- lecture. With the continuous improvement of deep tabular models on established benchmarks (McElfresh et al., 2023; Ye et al., 2024), the practical deployment of such models has become pressing consideration.
We further investigate the impact of temporal shifts on deep tabular methods from the perspective of feature representations. Through the visualization of the model's deep-layer representations, we observe that the trends and periodie information, which are prevalent in the raw data, gradually diminish in the feature representations. This indi- cates a loss of temporal information in the representation, leading to the restriction of the model's temporal prediction ability. This also explains why existing methods encounter challenges in addressing temporal shifts.
2.2. Distribution Shift in Tabular Data
Current research on distribution shifts can be broadly catego- rized into two main approaches. The first category focuses on scenarios in which target domain data is partially avail- able. In these cases, transfer learning techniques are com- monly employed to dynamically adjust model parameters during deployment by leveraging test-time data. TableShift (Gardner et al., 2023) applies various classical methods of domain generalization (Ganin et al., 2016) and domain adaptation (Sun & Saenko, 2016) into deep tabular learning, alongside the recently proposed test-time adaptation tech- niques for tabular data (Kim et al., 2024). While effective in certain contexts, these approaches often assume the avail- ability of target domain information at test time, which may be infeasible in real-world settings. In our experimental setup, the test time information is entirely invisible during both the training and testing stage, thus methods in this category are not applicable to our setting.
Based on the analysis presented above, it is essential to ef- fectively incorporate temporal information into the model. To address this gap. we develop a lightweight, plug-and- play temporal embedding method based on Fourier series expansion. This approach equips the model with learnable periodic and trend information while preserving its original capabilities. Experiments demonstrate that temporal embed- ding further enhances the capacity of all models to address shift problems. Furthermore, this method can be regarded as temporally adaptive approach. Once the model is pro- viddd with temporal information, it can acquire knowledge specific to different temporal stages. This capability allows the model to adjust adaptively after deployment.
In summary, this paper investigates the challenges posed by temporal distribution shifts in tabular data and explores effective strategies to address them. Starting with training protocol that fully leverages temporal data, we delve into the impact of temporal shifts during the model training process. We further propose a temporal embedding method to compensate for the loss of temporal information, thereby enhancing the model's ability to adapt to temporal shifts and improve its real-world performance. These insights offer crucial guidance for the future development of deep tabular methods in temporal shift scenario.
The second category addresses situations in which target domain data is entirely unavailable, representing a more common and challenging scenario. Approaches within this category can be further divided into two types: those aimed at enhancing model robustness and those focused on the active learning of shift patterns. The first type seeks to improve the inherent robustness and generalization of mod- els, thereby indirectly mitigating the impact of distribution shifts. For instance, Gorishniy et al. (2025) demonstrates the effectiveness of ensemble strategies in addressing distri- bution shifts in tabular data. Our exploration of the training protocol serves as effective approach to enhancing model generalization performance. The second type incorporates knowledge of distribution shifts directly into the model through adaptive methods. One such approach is presented by Helli et al. (2024), which employs second-order models to capture and adapt to shifts based on learned causal rela- tionships. However, methods in this category are heavily dependent on specific model architectures, such as PFN (Hollmann al., 2023), and tend to be computationally expensive. Consequently, we did not include a comparison with this category of methods. In contrast, our temporal embedding method offers a lightweight and plug-and-play solution for achieving temporal adaptability.
2. Related Work
2.1. Tabular Machine Learning
Tabular data is a common format across various applications, and the main solutions can be categorized into tree-based methods, token-based methods, retrieval-baat bassed methods, ensemble-based methods, and MLP architecture methods. Classical rree-based methods like Random Forest (Breiman, 2001), XGBoost (Chen& & Guestrin, 2016), LightGBM (Ke et al., 2017), and CatBoost (Prokhorenkova et al., 2018), remain powerful and widely adopted in real-world scenar- ios. FT-Transformer (FT-T) (Gorishniy et al.. 2021) is token- basen-based method that utilizes the transformer architec- ture, TabR (Gorishniy et al., 2024) and ModernNCA (Ye
3

---

Understanding the Limits of Deep Tabular Methods with Temporal Shift
Orittaal
Tisse
(Neeetage Chupse
2.1%
0,328
1.5%L
Gi Jay néo baee
2.24N
60 lag. Wi Nas
(4) wis leg wits Naw, reverned
0.313
6.22
Regroolert Difect for Vilrdccion
3300
3.11%
No Bair
ModecuNCA
Aucriage
Drogped
lag,
Figure 3. Left: Experimental design for temporal split strategies. The top panel shows the original baseline adopted by Rubachev et al. (2025). The middle panel (a)-(d) illustrates: (i) training lag (a vs. b; Section 4.1). (ii) validation bias (a vs. c; Section 4.2), and (ii) validation equivalence (b vs. d; Section 4.3). The bottom panel presents our proposed strategy (Section 4.4). Right: Performance improvement of different splitting strategies relative to split (c) on the TabReD benchmark, demonstrating benefits in reducing training lag and validation bias. Notably, the performance degradation from (b) (d) much smaller than the improvement achieved by (b), suggesting that adopting the alternative splitting strategy to maximizing data utilization is preferable. Detailed results in Appendix C.I.
While TableShift (Gardner et al., 2023) emphasizes domain- to-domain shifts, TabReD (Rubachev et al., 2025) introduces the concept of temporal shifts. They argue that all tabular data is inherently temporal, advocating for the use of tem- poral splits, especially in industrial applications where data is typically feature-rich and includes visible timestamps. In this study, we further investigate a training protocol and propose a more effective and robust framework for learning from temporal tabular data.
(e.g., Rd) and y denotes the label space (e.g., classes or real values). The goal is to learn a mapping f X - Y that can generalize from the observed data to unseen instances, effectively predicting ÿ = f(x) for a given input æ;,
The objective function f can be decomposed into two com- ponents: f = 9 o h. where g for feature extraction and h for prediction. In MLP architectures, both g and h are neural networks, with g consisting of multiple layers and h being the final output layer. In ensemble-ber -aaebe methods, multiple models ft = o h4 are trained, and the final prediction is obtained by averaging the outputs, which reduces vari- ance and improves generalization. Retrieval-based methods use a neural network for g and non- -arammeeeeii prediction method, such soft k-NN adopted in ModernNCA (Ye et al., 2025), for h, where predictions are based on the clos- est neighbors of "candidates" - instances from the training set that are mapped to the representation space and serve as potential reference points during prediction.
2.3. Distribution Shift in Other Domains
The study of distribution shifts originated in computer vi- sion, with early research primarily focusing on transfer learning techniques to address domain-to-domain shifts, including domain generalization (Blanchard et al., 2011; 2021), domain adaptation (Ganin & Lempitsky, 2015), and test-time adaptation ( (Wang et al., 2021a). Later, the focus expanded to encompass continual distribution shifts, with strategies for adapting models to sequential domain changes (Wang et al., 2022), as well as addressing recurring (Hoang et al., 2024) and non-i.i.d (Gong et al., 2022) temporal shifts. Wild-Time (Yao et al., 2022) explores real-world temporal shifts but translates them into Nomain-to-domall settings, neglecting temporal continuity. In comparison, these methods are primarily designed for images and most utilize transfer learning. Many image-based methods, when directly applied to tabular data, are considered to perform poorly (Gardner et al., 2023). This may be due to the more complex and diverse distribution shifts in tabular data, which involve greater temporal dependencies and continuity.
A temporal tabular dataset collected before Tirain for model training and deployment can be represented as Dirsinval = Up D., where D. = ((.,, ye, , denotes the set of n4 instances collected at time 4, each attached with its timestamp. Tirain is the time at which training is performed. After training, the model is deployed to an open environment and evaluated on Drest =
Ul> Dr. The training data 'D Derainval need to be further split into training and validation set in training stage. TabReD (Rubachev et al., 2025) adopts temporal split where the data divided at T Tal. such that Derain U Dr and D = -UrTaetSTaa Dr.
A temporal distribution shift refers a specific type of distribution shift where the data is collected sequentially over time, and the underlying data distribution evolves as time progresses. Formally, let X, and }/ denote the feature space and label space at time 4, respectively. For 2, E X4 and yt E Vr, a temporal distribution shift may involve
3. Preliminary
Generally, a tabular dataset with instance is represented as D = {(2,,) p where a; E X is the input feature and
y: E 3/ is the target label. Here, X' denotes the feature space
4

---

Understanding the Limits of Deep Tabular Methods with Temporal Shift
Sgilt tro w/ log, wy hls
Siplit leg, we
(bi: wio lig we
leg,
Timee
Toan
Mape Portayy
The loss distribution shows how the model's performance distributed across time slices under different validation splitiing strategies. The vertical axis represents the loss, where lower is better. Non-lagged splits (b) and (d) achieve better performance around Traa compared to lagged splits (a) and (c), while the higher-biased split (c) performs better training-available data but fails to generalize compared to the lower-biases split (a). The loss distribution is smoothed by a Gaussian filter for better visualization. Right: The MMD heatmap visualizing the distribution distance between different time slices using linear kernel. Time slices are divided by date.
Figure 4. Left:
changes in p(ane) * p(æe), , p(u)) * p(ye): p(yrr (ee), or even X, + Xr or 34. + Vir, for some i + V'
): p(ue
ax)) o
more instances closer to the test time for training. An intu- itive hypothesis is that instances closer to the test time are more reliable, as the distribution near the test time tends to be more similar to the actual test-time distribution, and using them for training could have a greater effect than for validation. Hence we adapt a pair of splitting strategies, shown in Figure 3 left (a) and (b), where identical validation and test sets are maintained while varying the lag between training and test set, thereby isolating and quantifying the impact of training lag on model performance.
Conventional approaches for tabular data analysis have pri- marily relied on the i.i.d. assumption, where the feature ex- traction function g typically maps all input data to a shared representation space, implicitly assuming that the data distri- bution remains stationary over time. As result, the impact of temporal dynamics the effectiveness of these methods remains largely unexplored and warrants investigation.
4. Why Temporal Split Ineffective?
We selected MLP-PLR, ModernNCA, and TabM as rep- resentative methods for MLP architecture, retrieval based, and ensemble- based methods, respectively, and conducted experiments of splitting strategy on these three method, Ex- perimental results shown in Figure right infers that this hypothesis holds for all three methods, with total average improvement of 1.62%. Among them, the retrieval based methods ModernNCA shows the highest improvement of 2. 19%, indicating the importance of no-lag candidates for retrieval-bar based methods in the presence of temporal shifts.
The fundamental challenge in addressing temporal shifts stems from the inherent difficulty characterizing evolving data distributions, which requires models to maintain robust- ness against unknown future variations. Temporal splits are frequently used in forecasting-related tasks to uphold the sequential dependencies inherent in the data (Bergmeir & Benítez, 2012: Zeng et al., 2023; Han et al., 2024), yet they generally show inferior performance compared to random splits in tabular data contexts, as evidenced in Section and Figure 2. This counterintuitive phenomenon warrants systematic analysis through multiple perspectives: We first start with the most intuitive distinction between random split and temporal split, and verify the hypothesis of how training lag (Section 4.1) and validation bias (Section 4.2) impact the model performance. Then we discover the equiv- alence of the validation set in different temporal directions (Section 4.3). Building on this, we propose an enhanced splitting strategy (Section 4.4) for temporally shifted tabular data, which achieves performance comparable to random splitting while offering improved stability.
4.2. Validation Bias
Current deep tabular methods rely on information from the validation set for model selection, as deep learning models are optimized epoch-wise during training. In the context of temporal shifts, this reliance becomes particularly prob- lematic, since there is lack of accurate validation of the test data in the training stage (Ganin & Lempitsky, 2015; Blanchard et al., 2021). The time gap between the training and test sets is larger than the gap between the training and validation sets in the previous temporal split, leading to a more significant distribution shift at test time. This makes it more challenging to accurately predict test-time instances compared to validation, thereby introducing bias into the validation process. Therefore we further design the splitting strategy shown in Figure 3 left (c), which shares the training and test sets with (a), but the latter has a more considerable
4.1. Training Lag
From the perspective of the training set, temporal split adopted in Rubachev et al. (2025) introduces temporal lag
between training and test set, while random split provides
5

---

Understanding the Limits of Deep Tabular Methods with Temporal Shift
Splits
MLP
PLR
FT-T
SNN
DCNv2
TabR
MNCA
TabM
XGBoost
CatBoost
LGBM
Avg. Imp.
Mean Performance
t
Random +4.30%
+0.73%
+3.76%
+1.38%
+2.11%
-2.00%
-2.53%
+1.51%
+1.79% +2.06%
+2.09%
+1.73%
+2.17% +2.18% %
Ours
+3.50%
+0.75%
+2.78%
+1.38%
+3.01%
-2.20%
+1.25% -2.99%
+2.37%
+2.14%
Standard Deviation!
+
Random +1.81% + +126%
+0.15%
+44.0%
+0.06%
+224%
-74.9%
+44.3%
+456%
+105%
+616% +8.59% -30.6%
+154%
Ours
+29.7% +50.9%
+5.59%
+1..8%
+8.86%
-82.2%
- -10.8%
+37.7%
-15.2%
+ +16.7%
Performance Rankings_ J.
Random
8,250
5.625
5,625
10.250
9.625
8.000
4.750
2.750
3.125
3.125
4,875
Ours
8.000
5.750
7.500
9,500
8.375
8.125
4.875
4.000
3.375
2.125
4.375
Table 1. Comparison of performance and stability between the random split in Figure 2 and our proposed temporal split in Section 4.4, measured by the average percentage change on the TabReD benehmark, along with the performance ranking of each method. "PLR," "MNCA," and "L.GMM" denote "MLP-PLR," "ModerNCCA," " and "LightGBM," respectively. The pereentage change represents the difference in the mean (higher is better) or the standard deviation (lower is better, indicating stability) of performance, relative to the original temporal split in Rubachev al. (2025), for each method. The results show that our temporal splitting strategy achieves performance comparable to the random split, while offering significantly better stability. The ranking change is minimal, with token-hased methods favoring the random split and tree-hased methods performing better with our temporal split. The comparison of methods under random and temporal splits is also plotted in Figure 2 and Figure 7. Detailed results are provided in Appendix C. with extended rankings.
validation bias since the time interval difference between train-val and train-test is much larger.
validation bias. On a deeper level, reducing the training lag concentrates the model's performance around time point closer to the test time, while precise validation ensures that the model's s performance can effectively generalize from this concentrated time point to the test period.
The performance comparison of split (a) and (c), shown in Figure 3 right, confirms that validation bias also has no- table impact on performance, especially for ensemble-based methods. The total average improvement is 0.59%, while the ensemble-based TabM show a more significant improve- ment of 0.83%. This explainable since the ensemble- based methods are robust to the training data quality by reducing the variance, but sensitive to the bias of validation.
In addition, another interesting observation in Figure 4 left is that the model in split (b) never meet the most former data splits during training and validation, but achieves a relatively well performance on it compared to the test splits. Building on the insight that the validation set is mainly for maintain- ing generalization performance, this finding inspired us to explore whether data from the opposite temporal direction could be an effective validation set, which is rely on the ussumption that the temporal shifts distributed uniformly across each time slices. We further use a heatmap to visu- alize the distribution distance between different time slices using Maximum Mean Discrepancy (MMD) (Gretton et al., 2006), shown in Figure 4 right. The MMD heatmap re- veals that all datasets exhibit temporal shifts characterized by trends and multiple periodic components. The regular diagonal stripes both indicate the presence of periodicity, and suggest that the sample distributions at identical time in- tervals are similar, which confirms the empirical uniformity of temporal shifts across time slices.
4.3. Bridging the Past and Future in Temporal Split
Through the above exploration, we have confirmed the im- provements in model performance achieved by reducing the training lag and validation bias. The main question now becomes how to utilize the above insights.
To illustrate how reducing the training lag and validation bias improves model performance, we visualize the loss distribution of the model across different time slices under different validation splitting strategies. Figure 4 left shows the loss distribution for the CT dataset using MLP-PLR model, it clearly shows how reducing training lag and val- idation bias improves the model performance. Each splits achieve the best performance among the training-available time slices, which allows the model to perform better on instances closer to the test time in the non-lagged split (b) compared to the lagged splits (a) and (c). Furthermore, the higher-biased split (c) works better on training splits but struggles on the test splits, while the lower-biased split (a) achieves a more balanced performance across training and test splits. This indicates that a lower-biased validation set brings a more precise direction of the test-time distribu- tion, enabling the model to generalize better. The above observation again enhances the insights of training lag and
Finally we design the splitting strategy shown in Figure left (d), where the validation set is in the opposite temporal direction compared to split (b), meanwhile maintaining the training lag and validation bias. The results in Figure 3 right indicate performance drop of 0.91% 1%% compared to split (b). However, it is important to highlight that, this performance degradation is noticeably smaller than the improvement observed in (b) relative to (a). This indicates that adopting this alternative splitting strategy to minimize the training gap is always desirable approach.
6

---

Understanding the Limits of Deep Tabular Methods with Temporal Shift
Searty Pesseiitty
B. Tane dr Bualy
Orsginal Decaset
MLP Seproseciatioon
MLNShmpeee Hepresentatar:
Figure 5. Left: Detailed MMD healmap of the HI dataset, illustrating both trend (lighter colors farther from the diagonal) and yearly/weekly periodicity (stripes at different scales) in the data. Middle and Right: MMD heatmaps of the representations learned by an MLP before and after applying our temporal embedding. Without temporal embedding. the model captures only coarse- soasse-raiine patterns (e.g., weekday weekend) with weak discrimination. After incorporating temporal embedding. the learned representations aliga with the data distribution, capturing phase-specific temporal feature (e.g., day of the week) and achieving clear distinction. See Appendix C.4 for more results.
In detail, the model performance this splitting strategy is highly dependent the dataset, which is primarily due to the nonuniform temporal sampling in some datasets, where maintaining the same amount of validation data compro- mises the consistency of the time intervals. Further explana- tion can be found in Appendix A.
split results in a much larger increase of 153.81%. This indi- cates that, while both methods achieve similar performance gains, our temporal split significantly outperforms the ran- dom split in terms of stability. A comparison of stability using the robustness score is provided in Appendix C.I.
5. What is Lost in Temporal Training?
4.4. Our Temporal Split
Looking back to the MMD heatmap in Figure right, we ob- serve that the original data distribution offers a rich source of temporal information, including the periodicity and the trend. Figure 5 left presents detailed MMD heatmap visu- alization of the HI dataset, revealing both yearly and weekly periodicity, as well as the underlying trend. We further investigate the impact of temporal shifts on deep tabular methods from the perspective of feature representations.
Based on the above findings, we introduce the following training protocol for temporal tabular data:
1. The lag between training and test should be mini- mized since instances near the test time more valu- able for training, rather than for model selection.
2. The validation bias should be minimized, which can be achieved by reducing the time interval difference be- tween train-val and train-test.
Unexpectedly, by comparing the MMD heatmaps of the learned representations of MLP in our training protocol (shown in Figure 5 mid), observe that the periodicity and the trend are lost in the model representations, Instead of the diagonal stripes observed in the original data distribu- tion, the learned representations exhibit only shallow grids and a more uniform distribution, suggesting that temporal information has not been effectively preserved. This indi- cates that the model has captured the distinction between weekdays and weekends but failed to capture the long-term periodicity and finer details of short-term cycles.
The equivalence property of the validation set in different temporal direction maintained for most tasks. An effective validation is available in the opposite temporal direction by aligning the degree of shift in the validation set with the actual shift between training and testing data.
We further propose a more effective temporal splitting strat- egy that fully leverages this protocol, shown in the bottom of Figure left. this strategy, the training lag minimized to zero, and the validation set, as it has a similar time interval relative to the training set thus exhibits similar degree of distribution shift.
is also aligned with the test
This phenomenon may account for the suboptimal perfor- mance of datasets with clear periodic patterns, such as CT, DE, and MR datasets, which are socially related and ex- hibit distinct weekly cycles, while the WE dataset, being influenced by natural cycles, shows a clear yearly pattern. We would expect models using temporal splits to capture long-term periodicity and trends, enabling them to learn extrapolative knowledge. However, this critical knowledge does not seem to be effectively learned. In contrast, random splits appear more proficient at capturing local patterns, par-
The performance and stability comparison between the ran- dom split and our newly proposed temporal split is presented in Table 1. The results demonstrate that our splitting strat- egy achieves a performance improvement comparable to the random split (2.1 18% vs. 2.17% on average) relative to the baseline temporal split in Rubachev et al. (2025). Addition- ally. while our temporal split shows modest increase in the standard deviation of performance scores (AUC for classi- fication and RMSE for regression) by 16.69%, the random

---

Understanding the Limits of Deep Tabular Methods with Temporal Shift
Daccant Temponul Info
based approach, the model can learn temporal stage- specific knowledge by leveraging timestamp information, thereby adaptively adjusting its mapping at different tempo- ral stages after deployment. Formally, the model's objective function can be written as f = o ht where maps the input features to the same representation space. When the data exhibits temporal shifts, this implies that there exists a specific mapping ft = 9: h4 at time step t. The model can now learn phase-specific knowledge and adjust the mapping ft accordingly, as illustrated in Figure 6.
Yeuts
Time
Taxt
Lanarigh Tamponal Info
Time
In our embedding method, we fit multi-scale periodicity using Fourier series expansion (Tancik et al., 2020; Li et al., 2021). The Fourier series can be expressed as:
Figure 6. Illustration of adaptive approaches for temporal shift mitigation. Our proposed temporal embedding method offers lightweight alternative that implicitly enables adaptation by em- bedding temporal information. This allows the model to learn phase-specific knowledge and adjust the mapping fo accordingly, thereby achieving temporal generalization.
12 T
(2xkt T
f(t)
ag sin
+
by cos
By increasing the order of the expansion, we can approx- imate any continuous periodic function f with period T according the approximation properties of Fourier series. Our temporal embedding can be described as follows:
Emb.
MLP
MLP-PLR
ModemNCA
Avg. Imp.
Num -0.04%
-0.06%
-0.04%
-0.05%
Time -0.70%
-0.15%
-0.32%
-0.39%
PLR
+0.70%
+0.01 %
+0.02%
+0.25%
Ours
+1.31 %
+0001 %
+..30%
+0.54%
(t) = [ReLU (Linear (Periodic(t))). [r)nd
Table 2. Performance gain with temporal embeddings on the TabReD benchmark. Embeddings are applied only to timestamps, which are then treated numerical features for the model. Non- learnable embeddings yield limited improvement: the learnable PLR variant helps MLP slightly. Our temporal embedding consis- tently outperforms all others. See Appendix C.2 for details.
where is the timestamp, and the two components of the embedding each capture the periodicity and trend of the timestamp. The former part is further defined as
Periodic(t) = [Fourier(t, Fi)....,Fourierrf Tm)];
where T; are given periodicity priors for Fourier-based embedding. In our experiments, we set Ti to represent yearly, monthly, weekly, and daily periodicities. These choices reflect common temporal patterns observed in the datasets. For example, in datasets with natural temporal correlations such as WE, yearly and daily periodicities are effective, while datasets like HI and CT benefit from model- ing socially driven patterns captured by monthly and weekly periodicities. Pre-defined periodicity priors yield more sta- ble and interpretable results, as illustrated in Appendix C.2. Each Fourier-basee embedding is defined as
ticularly those associated with short-term periodicity. This could explain why temporal splitting does not consistently outperform random splitting in these cases. Furthermore, it also explains the reasons why existing methods encounter challenges in dealing with temporal shifts.
6. Temporal Embeddings
Building on the above analysis, it is essential to incorpo- rate temporal information into the model in a manner that effectively captures the underlying temporal dependencies. To address this issue, we propose lightweight, plug-and: play temporal embedding method specifically designed for timestamps, aiming to investigate whether providing explicit temporal information through embedding can lead to per- formance improvements. Empirically, timestamps are often treated as noise and discarded. However, in the context of temporal shifts and temporal splitting, timestamps likely contain crucial temporal information that enables the model to align with the periodicity and trends inherent in the tem- poral sequence. Existing works have also emphasized that, in certain contexts, timestamps serve as valuable feature (Wang et al., 2024; Zeng et al., 2024).
2 T
27 }
Fourier(t,TT sin
COS
R2K
T
for orders k E (1. K}. representing the K-term Fourier series expansion. We assign the Fourier orders individually for each periodic prior. By combining mul- tiple periodic components and projecting them through a learnable linear layer, the embedding approximates the ag- gregation of Fourier coefficients ag and by as formulated in Equation (1), A subsequent ReLU activation promotes spar- sity and emphasizes salient frequencies, thereby improving the quality of the learned periodic representations.
In addition to the periodic component, we also provide an optional trend term for the temporal embedding. When the
Our temporal embedding also serves as an adaptive model-
8

---

Understanding the Limits of Deep Tabular Methods with Temporal Shift
Our Training Protocol
With Temporal Embedding
5.41%
2,30%
Change
5.41%
10.05
4.67%
4.1%
4.1%
4.39%
0.04
Forrs
#037M
6.73%
0.003
2,05
2.31%
0.02
1.22%
5.0%
0.01
MEP
MUAPLL
XCR00s:
Catéloos:
Light
SNN
DON2
Thas
ModeraNCA
Tbbbi
MLP
MUP-PLR
LightGBM
SOND
DON22
FT:1
ThbR
ModeraNCA
MLP Architecture
Trep Based
Token Based
Retrieval Based
Rasemble Based
Improvement
Figure 7. Performance comparison before and after adopting our temporal embedding into our training protocol on the TabReD benchmark. This figure follows the same setup as Figure 2, allowing for direct comparison. Detailed results are provided Appendix
trend is enabled, the final embedding is augmented with a standardized timestamp, which captures the linear temporal shift beyond the periodic components, represented by
require dedicated embedding strategies rather than relying existing numerical embedding approaches. After ap- plying temporal embedding, both ModemnNA and TabR demonstrate strong performance, indicating that with an appropriate training protocol and temporal embedding, even retrieval-basee methods, which are typically most affected by distributional shift, can regain their practical utility.
(t) z-score(t) R.
To thoroughly evaluate the effectiveness of our embedding method, we designed series of comparative experiments, including the following configurations:
The MMD heatmap of the model representation after adopt- ing temporal embedding is shown in Figure right and Appendix C.4. The patterns are closer to the original data, reflecting that it captures correct temporal information, thus effectively alleviating the loss of temporal information dur- ing training. The reappearance of diagonal stripes indicates that the model has learned independent representations for each temporal phase within the period, thereby confirming the adaptive role of temporal embedding.
None: Timestamps are not utilized as the baseline.
Num: Treating the timestamp as a single numerical fea-
ture and directly inputting into the model.
Time: Decomposing the timestamp into six numerical fea- tures: year, month, day, hour, minute, and second, which introduces partial periodicity information while maintain- ing human-readable representation of the timestamp. PLR: Applying PLR embedding (Gorishniy et al., 2022) to the timestamp, treating the resulting representations as numerical features capture temporal patterns adaptively.
7. Conclusion
We choose MLP, MLP-PLR and NCA for compari- son, the results are presented in Table 2. The results indicate that while directly using non-learnable embedding meth- ods can be effective in certain cases, it generally leads to a performance degradation, which is consistent with the common practice of treating timestamps as noise. The per- formance of PLR embedding is not stable, likely because it discards linear trends and struggles accurately capture periodic patterns. Results from the MLP-PLR method show no significant improvement, which may be attributed to its incompatibility with the existing numerical embedding.
In this paper, we first investigate the challenges posed by temporal distribution shifts in tabular data, with focus effective strategies for addressing them. Starting with a training protocol that fully leverages temporal data, we analyze the impact of training lag, validation bias, and the equivalence of validation. Building on these insights, we propose novel splitting strategy that significantly improves model performance. We further demonstrate that captur- ing temporal information during training is crucial, and observe that periodic and trend information is often lost in the learned model representations. To compensate for this loss, we introduce a temporal embedding method that incor- porates temporal information from timestamps, improving the model's '' adaptability to temporal shifts. By combining the temporal split with the proposed embedding, we observe marked improvements in model performance, par- ticularly for retrieval- based models that previously struggled under temporal shifts. These findings provide valuable in- sights for advancing deep learning approaches for temporal tabular data, highlighting the importance of both temporal data training protocol and temporal feature integration.
The models' performance improvement after adopting our temporal embedding presented in Figure 7. Most meth- ods demonstrate improvements, highlighting the importance of leveraging temporal information in addressing temporal shift tasks. As previously mentioned, ILP-PLR and TabM exhibit limited gains, likely due to their incompatibility with PLR embedding. However, these models achieve substan- tially greater improvements when the temporal embedding is directly integrated into the model backbone, as shown in Appendix C.2. This suggests that temporal features may

---

Understanding the Limits of Deep Tabular Methods with Temporal Shift
Acknowledgement
Gardner, J., Popovic, Z., and Schmidt, L. Benchmarking dis- tribution shift in tabular data with tableshift. In NeurlPS, pp. 53385-53432, 2023.
This work is partially supported by National Key R&D Pro- gram of China (224YYE02008000, NSFC (62376118), Key Program of Jiangsu Science Foundation (BK20243012), Collaborative Innovation Center of Novel Software Tech- nology and Industrialization.
Gong, T., Jeong, J., Kim, T., Kim, Y., Shin, J., and Lee, S.-J. NOTE: robust continual test-time adaptation against tem- poral correlation. In NeurlPS, pp. 27253 2022.
Gorishniy, Y., Rubachev, I., Khrulkov, V., and Babenko, A. Revisiting deep learning models for tabular data, In NeurlPS, pp. 18932- 18943, 2021.
Impact Statement
This paper aims to advance the field of machine learning by addressing the critical challenge of temporal distribu- tion shifts in tabular data, which frequently occur in real- world applications. The proposed temporal training protocol and temporal embedding method offer practical improve- ments for deploying existing tabular models in open envi- ronments.
Gorishniy, Y., Rubachev, I., and Babenko, A. On embed- dings for numerical features in tabular deep learning. In NeurIPS, pp. 24991-25004, 2022.
Gorishniy, Y., Rubachev, I., Kartashev, N., Shlenskii, D., Kotelnikov, A., and Babenko, A. Tabr: Tabular deep learning meets nearest neighbors. In ICLR, 2024.
References
Gorishniy, Y., Kotelnikov, A., and Babenko, A. Tabm: Advancing tabular deep learning with parameter-efficien ensembling. In ICLR, 2025.
Akiba, T., Sano, S., Yanase, T., Ohta, T., and Koyama, M. Optuna: A next-generation hyperparameter optimization framework. In KDD, pp. 2623 2631. ACM, 2019.
Gretton, A., Borgwardt, K. M., Rasch, M. Schölkopf, B., and Smola, A. J. A kernel method for the two-sample- problem. In NIPS, pp. 513-520, 2006.
Arik, Ô. and Pfister, T. Tabnet: Attentive interpretable tabular learning. In AAAI, pp. 6679 6679-6687, 6687, 2021.
Han, L., Ye, H.-J., and Zhan, D.C.. The capacity and ro- bustness trade-off: Revisiting the channel independent strategy for multivariate time series forecasting. IEEE Trans. Knowl. Data Eng.. 36:7 129 7142, 2024.
Bergmeir, C. and Benítez, J. M. On the use of cross- validation for time series predictor evaluation. Inf. Sci., 191:192-213, 2012.
Bishop, C. M. Pattern recognition and machine learning, 5th Edition. Information science and statistics. Springer, 2007.
Hassan, M. R., Al-Insaif, S., Hossain, M. I., and Kamruz- zaman, machine learning approach for prediction of pregnancy outcome following IVF treatment. Neural Comput. Appl., 32:2283-2297, 2020.
Blanchard, G., Lee, G., and Scott, C. Generalizing from several related classification tasks to a new unlabeled sample. In NIPS, pp. 2178-2186 2011.
Helli, K., Schnurr, D., Hollmann, N., Müller, S., and Hutter, F, Drift-resilient tabpfn: In-context learning temporal distribution shifts on tabular data. NeurlPS, pp. 98742- 98781, 2024.
Blanchard, G., Deshmukh, A. A., Dogan, Ü. Lee, G., and Scott, C. Domain generalization by marginal transfer learning. J. Mach. Learn. Res., 22:1-55, 2021.
Hoang, T., Vo, M., and Do. M. Persistent test-time adap- tation in recurring testing scenarios. In NeuriPS, pp- 124402-123442, 2024.
Breiman, L. Random forests. Mach. Learn., 45:5-32, 2001.
Hollmann, N., Müller, S., Eggensperger, K., and Hutter, F. Tabpfn: transformer that solves small tabular classifi- cation problems in second. In ICLR, 2023.
Chen, T. and Guestrin, C. Xgboost: A scalable tree boosting system. In KDD, pp. 785-794, 2016.
Ganin, Y. and Lempitsky, V. S. Unsupervised domain adap- tation by backpropagation. In ICML, pp. 1180-1189, 2015.
Hollmann, N., Müller, S., Purucker, L., Krishnakumar, A.,
Körfer, M., Hoo, S. B.. Schirmeisterr R. T., and Hutter, F. Accurate predictions on small data with tabular foundation model. Nature, 637:319-3266 2025.
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., and Lempitsky, S. Nommin--dversaae training neural networks. J. Mach. Learn. Res., 155::55 2016.
Holzmüller, D., Grinsztajn, L., and Steinwart, I. Better by default: Strong pre-tuned mlps and boosted trees on tabular data. In NeurlPS, pp. 26577-26658, 2024.
10

---

Understanding the Limits of Deep Tabular Methods with Temporal Shift
Jiang, J.-P., Ye, H.-J., Wang, L., Yang, Y., Jiang, Y., and Zhan, D.-C. Tabular insights, visual impacts: transferring expertise from tables to images. In ICML, pp. 21988- 22009, 2024.
e-commerce stores. Decis. Support Syst., 59:296-311, 2014.
Prokhorenkova, L. O., Gusev, G., Vorobev, A., Dorogush, A. V., and Gulin, A. Catboost: unbiased boosting with categorical features. In NewrIPS, pp. 6639-6649, 2018.
Jiang. J.-P., Liu, S.-Y., Cai, H.-R., Zhou, Q, and Ye, H.-J. Representation learning for tabular data: A comprehen- sive survey. CoRR, abs/2504.16109 2025.
Rubachev, I., Kartashev, N., Gorishniy, Y., and Babenko, A. Tabred: A benchmark of tabular machine learning in-the-wild. In ICLR, 2025.
Ke. G., Meng, Q.. Finley, T., Wang. T., Chen, W. Ma, W., Ye, Q., and Liu, T.-Y. Lightghm: A highly efficient gradient boosting decision tree. In NIPS, pp. 3146-3154, 2017.
Sun, B. and Saenko, K. Deep CORAL: correlation align- ment for deep domain adaptation. In ECCV Workshops (3), pp. 450, 2016.
Kim, C., Kim, T., Woo, S., Yang. J. Y., and Yang, E. Adaptable: Test-time adaptation for tabular data via shift- aware uncertainty calibrator and label distribution handler. CoRR, abs/2407.10784 2024.
Tancik, M., Srinivasan, P. P., Mildenhall, B., Fridovich-Kell S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J. T., and Ng. R. Fourier features let networks learn high frequency functions low dimensional domains. In NeurIPS, pp. 7537-7547, 2020.
Klambauer, G., Unterthiner, T., Mayr, A., and Hochreiter, S. Self-normalizing neural networks. In NIPS, pp. 971-980, 2017.
Thielmann, A. F., Kumar, M.. Weisser, C., Reuter, A., Säfken, B., and Samice, S. Mambular: A sequential model for tabular deep learning. CoRR, abs/2408.06291, 2024.
Li, Y., Si, S., G., Hsieh, C.-J, and Bengio, S. Learnalle fourier features for multi-dimensional spatial positional encoding. In NeurIPS, pp. 15816-15829 2021.
Liu, S.-Y. and Ye, H.J. Tabpfn unleashed: A scalable and effective solution to tabular classification problems. CoRR, abs/2502.00527,, 2025.
Tschalzev, A.. Marton, S., Lüdtke, S., Bartelt, C., and Stuck- enschmidt, H. A data-centric perspective on evaluating machine learning models for tabular data. In NeurIPS, pp. 95896-95930. 2024.
Liu, S.-Y., Cai, H.-R., Zhou, Q.-L., and Ye, H.JJ TAL- ENT: A tabular analytics and learning toolbox. CoRR, abs/2407.04057, 2024.
Wang, C., Qi, Q.. Wang. J., Sun, H.,, Zhuang, Z., Wu, J., and Liao, J. Rethinking the power of timestamps for robust time series forecasting: A global-local fusion perspective. In NeurlPS, pp. 22206-22232, 2024.
Liu, S.-Y. Zhou, Q. and Ye, H.JJ Make still further progress: Chain of thoughts for tabular data leaderboard. CoRR, abs/2505.13422 2025.
Wang. D., Shelhamer, E., Liu, S., Olshausen, B. A., and Darrell, Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021a.
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. In ICLR, 2019.
Wang. Q., Fink, O., Gool, L. V., and Dai, D. Continual test-time domain adaptation. In CVPR, pp. 7191-7201, 2022.
Matveev, A. and Sidorova, A. Sberbank russian housing market. https: /ttpss:/kaggle.. sherraank-russiaa--houisss maettt, 2017. Kaggle.
Wang, R., Shivanna, R., Cheng, D. Z., Jain, S., Lin, D., Hong, L., and Chi, E. H. DCN V2: improved deep & cross network and practical lessons for web-scale learning rank systems. In WWW, pp. 1785-1797, 2021b.
McElfresh, D. C., Khandagale, S., Valverde, J., C., V. P., Ramakrishnan, G., Goldblum, M., and White, C. When do neural nets outperform boosted trees on tabular data? In NewrIPS, pp. 76336-76369, 2023.
Yao, H., Choi, C., Cao, B., Lee, Y., Koh, P. W., and Finn, C. Wild-time: A benchmark of in-the-wild distribution shift over time. In NeurlPS, pp. 10309-10324, 2022.
Mohri, M., Rostamizadeh, A., and Talwalkar, A. Founda- tions of Machine Learning. Adaptive computation and machine learning. MIT Press, 2012.
Ye, H.-J., Zhou, Q.L.,, Yin, H.-H., Zhan, D.-C., and Chao, W.-L. Rethinking pre- pre-training in tabular data: neigh- borhood embedding perspective. CoRR, abs/2311 100005,5 2023.
Nederstigt, J., Aanen, S. S., Vandic, D., and Frasincar, F. FLOPPIES: framework for large-scale ontology population of product information from tabular data in

---

Understanding the Limits of Deep Tabular Methods with Temporal Shift
Ye, H.-J., Liu, S.-Y., Cai, H.-R., Zhou, Q.-.., and Zhan, D.-C. A closer look deep learning methods tabular datasets. CoRR, abs/2407 0.09660,............... 2024.
Ye, H.-J., Yin, H.-H., Zhan, D.-C., and Chao, W.-L. Re- visiting nearest neighbor for tabular data: deep tabular baseline decades later. In ICLR, 2025.
Zeng, A., Chen, M., Zhang, L., and Xu, Q. Are transformers effective for time series forecasting? In AAAI, pp- 11121 11128, 2023.
Zeng. C., Tian, Y., Zheng, G., and Gao, Y. How much can time-related features enhance time series forecasting? CoRR, abs/241 2024.
Zhou, Z.-H. Open environment machine learning. National Science Review, 9:nwac 2022.
12

---

Understanding the Limits of Deep Tabular Methods with Temporal Shift
A.
Dataset Explanation
The detailed information about the dataset is provided in Table 3. We adopt the full TabReD dataset (Rubachev et al., 2025) without modification.
Abbr.
Dataset
Samples
Features
Task Type
Task Description
HI
Homesite Insurance
260753
296
Classification
Insurance plan acceptance prediction
EO
Ecom Offers
160057
119
Classification
Predict whether a user will redeem an offer
HD
HomeCredit Default
381664
696
Classification
Loan default prediction
SH
Sberbank Housing
28321
387
Regression
Real estate price prediction
CT
Cooking Time
319986
195
Regression
Restaurant order cooking time estimation
DE
Delivery ETA
350516
225
Regression
Grocery delivery courier ETA prediction
MR
Maps Routing
279945
1026
Regression
Navigation app ETA from live road- -raph features Weather prediction (temperature)
WE
Weather
423795
98
Regression
Table 3. Overview of Datasets. Task descriptions from Rubachev et al. (2025).
In Figure 3, Table 1, and Table 2, we compare the average performance improvement for each method under different strategies (e.g. splitting or temporal embedding), specifically the percentage increase in AUC on classification datasets and the percentage decrease in RMSE regression datasets. In Figure 2 and Figure 7, we present comparison of performance across different training protocols and methods, where all results are reported as performance improvements relative to the MLP performance under the original split in Rubachev et al. (2025). Since performance improvements between methods can often be influenced by outliers, we apply robust average, excluding the maximum and minimum performance improvements across the eight datasets before calculating the mean.
The only distinction in the implementation is that TabReD employs different encoding methods during preprocessing for numerical and categorical features for each method-datasee pair. Specifically, it uses identity or noisy-quantile encoding for numerical features and one-hot or ordinal encoding for categorical features. To ensure a fair comparison, we reproduced the experiment from TabReD by removing numerical encoding and fixing categorical encoding to one-hot encoding where necessary. This adjustment essential for accurately evaluating the performance of the methods and for advancing future research on temporal embeddings.
It is also important to note that the HD dataset suffers from severe class imbalance, which makes it challenging for methods with limited feature extraction capabilities, such as MLP, SNN (Klambauer et al., 2017), and DCNv2 (Wang et al., 2021b), to perform well without additional numerical feature encoding. On this dataset, the AUC of the naive MLP (as well as SNN and DCNv2 methods) drops to approximately 0.55, in contrast to other methods that typically achieve an AUC above 0.80. This large discrepancy renders direct comparison uninformative. To mitigate this issue, we adopt a robust average when reporting the percentage improvement over MLP. Nevertheless, the insights on training protocols and temporal embeddings explored in this work still lead significant performance improvements on this challenging dataset, detailed in Appendix C.
MM
Figure 8. The temporal sampling distribution for TabReD datasets. The time slices are divided into half days (i.e., 12 hours).
13

---

Understanding the Limits of Deep Tabular Methods with Temporal Shift
At the same time, our experiment on the equivalence (Figure 3) of the validation set shows that the datasets whose experimental results seriously fail to meet the equivalence assumption, such as SH and EO, are confirmed to have the most serious nonuniformity in sampling. The temporal sampling distribution for each dataset is shown in Figure 8.
Instead of focusing on the time span between the training and validation sets, we concentrate on ensuring that the number of instances in both sets is equal across different splits, thereby enabling a fair comparison of the splitting strategies. When the temporal sampling distribution becomes highly non-uniform, the observation in Figure no longer holds, as relies on a constant time span for the validation set. Consequently, our insights regarding the equivalence of the validation sets are significantly affected in such situations. This indicates that the conclusion regarding the equivalence of the validation set on different temporal directions is actually stronger, especially for datasets with uniform sampling over time.
B. Experimental Setup
Our source code is now available at http:://jihbb.ccmmmm/MDDDAAaaaal
We adopt preprocessing, training, evaluation and tuning setup from Ye et al. (2024) and Liu et al. (2024). We tune hyper- parameters using Optuna (Akiba al., 2019), performing 100 trials for most methods (except FI-T and TabR for only 25, explained below) to identify the best configuration. The syper-parametes search space follows exactly the settings in Rubachev et al. (2025), and can be referred from our source code. The hyper-parameter search space of our proposed temporal embedding is summarized separately in Table 4. Using these optimal with 15 random seeds, and the average performance across seeds is reported. We label the attribute type (numerical or categorical) for gradient boosting methods such as CatBoost. For all deep learning methods, we use a batch size of 1024 and A (Loshchilov & Hutter, 2019) as the optimizer, with an early stopping patience of 16. For classification tasks, we evaluate models using AUC (higher is better) as the primary metric and use RMSE (lower is better) for regression tasks select the best-performing model during training the validation set.
each method is trained
Parameter
Distribution
year.order
2' Uniformint [, 7
month.order
2" UniformInt [,
day-order
2' UniformInt [1
hour.order
2' UniformInt [
trend
(True, False]
d embedding
2" UniformInt [1, 5]
Table 4. Temporal embedding hyper-paramette space.
We followed Rubachev et al. (2025) and performed only 25 hyper- pppre-paraeeter tuning trials for FT-T and TabR, as these methods exhibit lower efficiency on datasets with large feature dimensions and sample sizes. For our temporal embedding, we conducted separate hyper-parameter searches for the periodic order and linear trend, to selectively extract temporal information while reducing computational overhead. However, since 25 tuning trials were insufficient to identify the optimal parameerrsss for the temporal embedding, we performed global tuning of the temporal embedding order for FT-T and TabR, with all periodic components sharing the same order.
To ensure the validity of random splitting, each group of random split experiments was tested on three distinct random splits, with 15 random seeds run on each split. The mean performance across these runs is reported as the final result. The standard deviation of the random split is calculated based on all 45 results (3 splits x 15 seeds), as the random split is subject to variability from both the split selection and the running seeds during the training phase. This approach better reflects the overall stability of the standard procedure.
14

---

Understanding the Limits of Deep Tabular Methods with Temporal Shift
C. Additional Results
C.I. Splitting Strategies
Methods
Splits
HIt
E01
HD1
SH4
CT4
DEL
MR1
WEL
Avg. Imp.
Original
0.6782
0.5713
0.5872
0.6822
0.5107
0.5981
0.2330
2.2232
Mambular
Ours
0.6870
0.6010
0.5396
0,5492
0.5080
0,5999
0.2297
2.1972
+2.59%
Original
0.8215
0.5660
0.5000
0.2276
0.4858
0.5529
0.1673
1.7076
TabPFN v2
Ours
0.8205
0.5717
0.5000
0.2204
0.4804
0.5619
0.1740
1.6031
+0.71%
Table The performance comparison the autoregressive method Mambular (Thiclmann et al., 2024) the tabular general method TabPFN v2 (Hollmann et al., 2025) under original temporal split in Rubachev et al. (2025) and our refined temporal split. Despite the large dataset size leading to suboptimal overall performance, Mambular benefits significantly from our refined temporal split. TabPFN v2, since training required, modificd the context selection: 10,000 contexts were randomly chosen (Original) and the last 10,000 samples were selected as the context (Ours). The results also show a performance improvement.
To further validate the general applicability of training protocol, apply our temporal splitting strategy two new model families: the autoregressive model Mambular (Thielmann et al., 2024) and the tabular general model TabPFN-v2 (Hollmann et al., 2025). As shown in Table 5, our temporal split consistently improves performance compared to the original splits used by these models.
We further report detailed experimental results for the different data splitting strategies illustrated Figure 3, with corresponding metries presented in Table Specifically, Table 7 and Table compare the performance and standard deviation across three strategies: the baseline temporal split from Rubachev al. (2025), the random split discussed in Section 1, and our proposed temporal split described in Section 4.4. These results demonstrate the superior performance and stability of our refined temporal split. Furthermore, Figure 9 presents additional experiment using the robustness score to further highlight the consistency of our splitting strategy.
Methods
Splits
HIt
EO
HD1
SH
CTV
DEJ
MR
WEJ
Avg. Imp.
(c)
0.9484
0.5630
0,8426
0.2635
0.4828
0.5515
0.1629
1,5693
(a)
0,9575
0.5765
0.8470
0.2629
0.4826
0.5545
0.1630
1.5623
+0.51%
MLP-PLR
(b)
0,9590
0,5928
0.8481
0.2375
0.4811
0.5539
0.1630
1.5713
+2.09%
0.9476
0.5654
0.8451
0.2460
0.4814
0.5490
0.1637
1.5553
+1.06%
(c)
0,9469
0.5648
0.8424
0.2846
0.4846
0.5503
0.1637
1,5200
(a)
0.9564
0.5731
0.8444
0.2836
0.4832
0.5520
0.1634
1,5165
+0.44%
ModernNCA
(b)
0,9603
0.5898
0.8469
0.2430
0.4813
0,5514
0.1639
1.5258
+2.63%
(d)
0.9539
0.5615
0.8405
0.2469
0.4824
0.5533
0.1636
1.5391
+1.49%
(c)
0.9543
0.5756
0.8530
0.2725
0.4821
0.5502
0.1629
1.5210
(a)
0,9580
0.5809
0,8520
0.2587
0.4816
0.5546
0.1626
1,5080
+0.83%
TabM
(b)
0,9622
0.5996
0.8531
0.2448
0.4796
0.5543
0.1620
1.5263
+1.90%
(d)
0.9586
0.5772
0.8464
0.2457
0.4818
0.5482
0.1626
1.5165
+1.34%
Table 6. Detailed performance results (AUC for classification and RMSE for regression) of different splitting strategics illustrated in Figure left. These results are plotted Figure right.
15

---

Understanding the Limits of Deep Tabular Methods with Temporal Shift
Methods
Splits
HIT
EO1
HD1
SH.
CT4
DE1
MRJ
WEJ
Avg. Imp.
Original
0.9404
0.5866
0,4730
0.2802
0.4820
0.5526
0.1624
1.5331
MLP
Random
0.9383
0.6225
0.5532
0.2509
0.4814
0.5521
0.1619
1,5252
+4.30%
Ours
0.9360
0.6220
0.5508
0.2641
0,4821
0.5515
0.1619
1.5362
+3.50%
Original
0.9592
0.5816
0.8448
0.2412
0.4811
0.5533
0.1616
1.5185
MLP-PLR
Random
0.9599
0.6225
0.8208
0.2406
0.4800
0.5507
0.1616
1,5097
+0.73%
Ours
0.9596
0.6185
0,8166
0.2361
0.4799
0.5481
0.1616
1,5235
+0.75%
Original
0.9562
0.5791
0.5301
0.2600
0.4814
0.5534
0.1627
1,5155
FT-T
Random
0.9616
0.6268
0.5846
0.2369
0.4804
0.5503
0.1622
1,5001
+3.76%
Ours
0.9591
0.6159
0.5746
0.2438
0.4807
0.5503
0.1623
1.5146
+2.78%
Original
0,9538
0,5795
0.5266
0.3292
0.4834
0.5543
0.1656
1,5604
SNN
Random
0,9535
0.6187
0.5358
0.3228
0.4829
0.5543
0.1648
1.5600
+1.38%
Ours
0.9480
0.6209
0.5591
0.3367
0.4825
0.5541
0.1648
1.5611
+1.38%
Original
0.9519
0.5846
0.5082
0.3425
0.4827
0.5519
0.1628
1,5305
DCNv2
Random
0.9486
0.6195
0.5484
0.3340
0.4814
0.5516
0.1623
1.5264
+2.11%
Ours
0.9460
0.6244
0.5505
0.3129
0.4824
0,5521
0.1622
1,5216
+3.01%
Original
0.9527
0.5727
0.8442
0.2676
0,4818
0.5557
0.1625
1 1.4782
TabR
Random
0.9543
0.6206
0,8147
0.2384
0.4880
0,5548
0.1622
1,4629
+2.00%
Ours
0.9605
0.6148
0.8342
0.2370
0,4883
0.5550
0.1623
1,4732
+2.20%
Original
0.9571
0.5712
0.8487
0.2526
0.4817
0.5523
0.1631
1,4977
ModernNCA
Random
0.9617
0.6246
0,8399
0.2299
0.4806
0.5510
0.1621
1,4773
+2.53%
Ours
0.9610
0.6341
0.8378
0.2325
0.4804
0.5520
0.1619
1.4857
+2.49%
Original
0.9590
0.5952
0.8549
0.2465
0.4799
0.5522
0.1610
1,4852
TabM
Random
0.9629
0.6332
0.8282
0.2305
0.4794
0.5495
0.1607
1,4681
+1.51%
Ours
0.9640
0.6325
0.8290
0.2306
0.4813
0.5500
0.1612
1.4887
+1.25%
Original
0.9388
0.5731
0.8174
0.2560
0.4879
0.5587
0.1744
1.7465
Linear
Random
0.9397
0.5895
0.8235
0.2458
0.4867
0.5591
0.1685
1.7425
+1.44%
Ours
0.9388
0.5944
0.8231
0.2435
0.4864
0.5596
0.1680
1.7464
+1.64%
Original
0.9609
0.5764
0.8627
0.2475
0.4823
0.5459
0.1616
1.4699
XGBoost
Random
0.9625
0.6200
0.8452
0.2298
0.4806
0.5468
0.1611
1.4566
+1.79%
Ours
0.9625
0.6199
0.8644
0.2262
0.4792
0.5520
0.1610
1.4700
+2.06%
Original
0.9612
0.5671
0.8588
0.2469
0.4824
0.5464
0.1619
1.4715
CatBoost
Random
0.9639
0.6213
0,8580
0.2340
0.4805
0,5471
0.1613
1,4556
+2.09%
Ours
0.9639
0.6242
0.8620
0.2292
0.4792
0.5495
0.1610
1,4654
+2.37%
Original
0.9600
0.5633
0.8580
0.2452
0.4826
0.5474
0.1618
1.4723
LightGBM
Random
0.9616
0.6136
0.8334
0.2322
0.4807
0.5469
0.1616
1.4471
+1.73%
Ours
0.9631
0.6164
0.8599
0.2260
0.4877
0.5500
0.1612
I 1.4654
+2.14%
Original 0.9537
0.5755
0.7971
0.2623
0.4870
0.5565
0.1653
1.5839
RandomForee Forest
Random
0.9580
0.6254
0.8142
0.2427
0.4846
0.5588
0.1649
1.5694
+2.50%
Ours
0.9580
0.6068
0.8171
0.2400
0.4841
0.5588
0.1647
1.5845
+2.17%
Table Detailed performance results (AUC for classification and RMSE for regression) of different splitting strategies the TabReD benchmark: the original temporal split from Rubachev et al. (2025), the random split in Section 1, and our proposed temporal split in Scction 4.4, These results are plotted in Figure 2 left, Figure 2 right, and Figure 7 left, respectively, while also illustrated in Table 1.
16

---

Understanding the Limits of Deep Tabular Methods with Temporal Shift
Methods
Splits
HIf
EOT
HDf
SH4
CT4
DEL
MRJ
WEL
Avg. Imp.
Original
0.0026
0.0033
0.0006
0.0281
0.0005
0.0012
0.0001
0.0050
MLP
Random
0.0042
0.0031
0,0011
0.0119
0.0004
0.0006
0.0001
0.0059
+1.81%
Ours
0.0053
0.0040
0.0015
0.0093
0.0006
0.0012
0.0001
0.0059
+29.68%
Original
0.0005
0.0035
0.0028
0.0043
0.0004
0.0014
0.0001
0.0046
MLP-PLR
Random
0.0014
0.0055
0.0102
0.0131
0.0003
0.0010
0.0002
0.0163
+12..77%
Ours
0.0004
0.0072
0.0084
0.0026
0.0003
0.0011
0.0003
0.0038
+50.91%
Original
0.0092
0.0061
0.0278
0.0142
0.0004
0.0028
0.0005
0.0058
FT-T
Random
0.0036
0.0095
0.0391
0.0082
0.0007
0.0014
0.0002
0.0084
+0.15%
Ours
0.0048
0.0129
0.0319
0.0119
0.0008
0.0011
0.0003
0.0051
+5.59%
Original
0.0006
0.0109
0.0166
0.0709
0.0009
0.0029
0.0003
0.0034
SNN
Random
0.0014
0.0053
0.0065
0.0776
0.0009
0.0023
0.0005
0.0132
+-3.95%
Ours
0.0011
0.0067
0.0154
0.1038
0.0007
0.0020
0.0005
0.0054
+16.77%
Original
0.0013
0.0061
0,0239
0.0810
0.0003
0.0006
0.0002
0.0068
DCNv2
Random
0.0020
0.0066
0.0160
0.0660
0.0004
0.0005
0.0001
0.0094
+0.06%
Ours
0.0019
0.0060
0.0158
0.0658
0.0007
0.0009
0.0001
0.0040
+8.66%
Original
0.0024
0.0065
0.0041
0.0170
0.0003
0.0016
0.0005
0.0062
TabR
Random
0.0052
0.0055
0.0165
0.0068
0.0037
0,0033
0.0003
0.0067
+224.06%
Ours
0.0009
0.0095
0.0044
0.0045
0.0017
0.0040
0.0004
0.0076
+ 82.17%%
Original
0.0060
0.0037
0.0018
0.0089
0.0007
0.0018
0.0001
0.0052
ModemnNA
Random
0.0007
0.0110
0.0074
0.0037
0.0005
0.0015
0.0004
0.0101
+74.89%
Ours
0.0009
0.0030
0.0059
0.0033
0.0005
0.0007
0.0001
0.0034
100779%
Original
0.0018
0.0057
0.0024
0.0092
0.0007
0.0014
0.0001
0.0046
TabM
Random
0.0014
0.0030
0.0128
0.0033
0.0007
0.0013
0.0002
0.0047
+44.34%
Ours
0.0002
0.0042
0.0080
0.0030
0.0010
0.0014
0.0004
0.0049
+37.66%
Original
0.0005
0.0058
0.0008
0.0134
0.0004
0.0008
0.0107
0.0031
Linear
Random
0.0009
0.0040
0.0018
0.0072
0.0003
0.0004
0.0073
0.0029
+4.96%
Ours
0.0009
0.0165
0.0006
0.0007
0.0002
0.0008
0.0008
0.0014
-859%
Original
0.0002
0.0005
0.0005
0.0004
0.0001
0.0001
0.0000
0.0008
XGBoost
Random
0.0003
0.0024
0.0091
0.0016
0.0002
0.0002
0.0001
0.0064
+456.05%
Ours
0.0001
0.0005
0.0002
0.0002
0.0001
0.0000
0.0001
0.0011
15.20%
Original
0.0003
0.0110
0.0005
0.0014
0.0001
0.0002
0.0000
0.0016
CatBoost
Random
0.0008
0.0025
0,0022
0.0017
0.0003
0,0002
0.0001
0.0021
+104.619 %
Ours
0.0002
0.0018
0.0004
0.0007
0.0001
0.0001
0.0000
0.0011
-30.61%
Original
0.0002
0.0008
0.0004
0.0004
0.0001
0.0002
0.0000
0.0013
LightGBM
Random
0.0005
0.0045
0.0081
0.0015
0.0003
0.0003
0.0004
0.0042
+616.129
Ours 0.0001
0.0008
0.0006
0.0002
0.0002
0.0001
0.0000
0.0012
+8.59%
Original 0.0001
0.0008
0.0008
0.0006
0.0001
0.0001
0.0000
0.0004
RandomForest
Random
0.0002
0.0029
0.0017
0.0024
0.0001
0.0003
0.0000
0.0004
+134.95%
Ours
0.0001
0.0004
0.0007
0.0004
0.0001
0.0001
0.0000
0.0004
-7.65%
Table Detailed standard deviation of performance results (AUC for classification and RMSE for regression) when adopting different splitting strategies on the TabReD benchmark: the original temporal split from Rubachev al. (2025), the random split in Section 1, and our proposed temporal split Section 4.4, These results illustrated Table
17

---

Understanding the Limits of Deep Tabular Methods with Temporal Shift
L20%
10% CI
Lohh
Ingremeeri MEA
Perkrrancd TeproweEat snder Sraat Tuedity
LATS
16%
Figure 9. Comparison of model performance and stability of our splitting strategy and random split using robustness score. From top to bottom, the figure illustrates the mean performance improvement baseline temporal split adopted in Rubachev al. (2025) under penalty, mild penalty, and strict penalty, where the first panel demonstrates the performance, while the latter two highlight the robustness.
The thorough comparison of the original temporal split in (Rubachev et al., 2025), the random split in Section 1, and our newly proposed temporal split in Section 4.4 demonstrated Figure Our proposed splitting strategy results a comparable performance to random split while offering better stability. Since random split suffers severe instability shown in Table 1, including the splitting selection and the running seed, which cannot be estimated the training stage, robustness score for comparison, which is defined as
RS = μ Kor
where μ1, a are the average and the standard deviation of model performance. We employed no penalty, mild penalry, and strict penalty, corresponding
0, & 1, and 2, respectively. We observed that without penalty, our temporal split performs comparably to the random split (2.18% vs. 2.17% on average), and as the penalty increases, the advantage of our splitting strategy becomes more pronounced (2.43% vs. 2.21 average). This indicates that our method not only achieves strong performance but also exhibits superior robustness.
C.2. Temporal Embeddings
We conducted further analysis the incompatibility between temporal embedding and PLR embedding discussed in Section 6. Our temporal embedding converts timestamps into numerical inputs, which may be incompatible with PLR numerical embedding (Gorishniy et al., 2022). Specifically, once timestamps are embedded, their representation reflects temporal similarity. Applying another periodic transformation via PLR could increase optimization difficulty. Directly feed the temporal embedding into the model backbone consistently improves, as demonstrated Table 9.
Methods
Emb.
HI1
EO
HD1
SH
CT
DE1
MR
WE|
Avg. Imp.
None
0.9596
0.6185
0.8166
0.2361
0.4799
0,5481
0.1616
1.5235
MLP-PLR
+Ours
0.9607
0.6110
0.8158
0.2338
0.4803
0.5494
0.1617
1.5133
+0.01%
Ours*
0.9609
0.6215
0.8155
0.2337
0.4788
0.5492
0.1616
1.5152
+.266%
None 0.9640
0.6325
0.8290
0.2306
0.4813
0.5500
0.1612
1.4887
TabM
+Ours
0.9629
0.6271
0.8363
0.2321
0.4791
0.5488
0.1609
1.4812
+0.07%
Ours*
0.9633
0.6260
0.8368
0.2297
0.4775
0.5490
0.1607
1.4873
+0.20%
None 0.9610
0.6341
0.8378
0.2325
0.4804
0.5520
0.1619
1.4857
ModernNCA
+ Ours 0.9620
0.6356
0.8316
0.2255
0.4791
0.5535
0.1617
1.4903
+0.30%
Ours*
0.9623
0.6357
0.8344
0.2262
0.4790
0.5515
0.1616
1.4855
+0.41%
Feed temporal embedding directly to the backbone.
Table 9. The performance comparison between the temporal embedding used in our paper and directly feeding the temporal encoding into the model backbone. All three methods show improvement, indicating that there may incompatibility between the temporal embodding and numerical embedding.
18

---

Understanding the Limits of Deep Tabular Methods with Temporal Shift
We also experimented with tuning hyperparameeerrs to find the cycles, results in Table 10. When using fixed prior periods, ModernNCA achieved 0.30% performance improvement, while setting adjustable cycles resulted in a -2.48% performance decline. In temporal distribution shift scenarios, due to the absence of an entirely accurate validation set, we believe that prior knowledge of fixed cycles is more stable and interpretable than adjustable cycles. It's also important to note that in many tasks, complete cycles are not available. For example, in the WE dataset, there is yearly cycle, but the training set does not span full year, which highlights the importance of prior knowledge.
Methods
Cycles
HIT
EOT
HDt
SH!
CT
DE
MR
WEJ
Avg. Imp.
None
0,9610
0.6341
0.8378
0.2325
0.4804
0.5520
0.1619
1.4857
ModemnCAA
Fixed
0.9620
0.6356
0.8316
0.2255
0.4791
0.5535
0.1617
1.4903
+0.30%
Adjustable
0.9575
0.5719
0.8355
0.2469
0.4820
0.5493
0.1630
1.4926
-2.20%
Table 10. When using adjustable eycles, the performance comparison with temporal information (none) and our temporal embedding (fixed) shows that ModernNCA experiences a performance drop of -2.48%, trailing behind the fixed cycle temporal embedding (+0.30%) This highlights that, temporal shift scenarios, tuning cycles based on the validation less reliable than using fixed prior cycles.
We further present the detailed performance results of different embedding methods discussed in Section 6 in Table 11, as well as the model performance before and after adopting our proposed temporal embedding in Table 12.
Methods
Emb.
HI
E0
HDt
SH
CT
DE
MR
WEL
Avg. Imp.
None
0.9360
0.6220
0.5508
0.2641
0.4821
0.5515
0.1619
1,5362
+ NNum
0.9350
0.6233
0.5505
0.2594
0.4802
0.5651
0.1617
1.5394
-0.4%
MLP
+ Time
0.9353
0.6241
0.5514
0.2435
0.5407
0.5640
0.1619
1.5272
-0.70%
+ +PLR
0.9482
0.6186
0.5509
0,2513
0.4814
0.5517
0.1621
1.5361
+0.70%
+ +Ours 0.9471
0.6252
0.5519
0.2431
0.4801
0.5518
0.1621
1.5319
+1.31%
None 0.9596
0,6185
0,8166
0.2361
0.4799
0.5481
0.1616
1,5235
+Num 0.9594
0.6248
0.8088
0,2353
0.4807
0.5524
0.1611
1.5270
0.06%
MLP-PLR
+ +ime 0.9593
0.6185
0.8303
0,2347
0.4811
0.5645
0.1615
1.5275
0.15%
PLR 0.9595
0.6179
0.8193
0.2331
0.4829
0.5536
0.1614
1.5212
+0.0%%
Ours 0.9607
0.6110
0.8158
0,2338
0.4803
0.5494
0.1617
1.5133
+0.01%
None 0.9610
0.6341
0.8378
0.2325
0.4804
0.5520
0.1619
1.4857
+Num 0.9616
0.6354
0.8267
0.2268
0.4804
0.5575
0.1630
1,4822
-0.44%
ModernNCA
+ Time 0.9614
0.6317
0.8384
0.2304
0.4819
0.5665
0.1625
1.4841
00.2%
+ PLL 0.9608
0.6334
0.8366
0.2299
0.4775
0.5599
0.1618
1.4854
+0.02%
+Ours
0.9620
0.6356
0.8316
0.2255
0.4791
0.5535
0.1617
1.4903
+0.30%
Table 11. Detailed performance results (AUC for classification and RMSE for regression) of different embedding methods discussed in Section within our training protocol the TabReD benchmark. The average improvements illustrated in Table 2.
19

---

Understanding the Limits of Deep Tabular Methods with Temporal Shift
Methods
Emb.
HIt
E
HDt
SH4
CT
DEL
MRJ
WF1
Avg. Imp.
None 0.9360
0.6220
0.5508
0.2641
0.4821
0.5515
0.1619
1.5362
MLP
+Ours
0.9471
0.6252
0.5519
0.2431
0.4801
0.5518
0.1621
1.5319
+1.31%
None 0.9596
0.6185
0.8166
0.2361
0.4799
0.5481
0.1616
1.5235
MLP-PLR
+ +Ours 0.9607
0.6110
0.8158
0.2338
0.4803
0.5494
0.1617
1.5133
+0.01%
None 0.9591
0.6159
0.5746
0.2438
0.4807
0.5503
0.1623
1.5146
FT-T
+ +Ours 0.9608
0.6211
0.5563
0.2363
0.4778
0.5503
0.1622
1,5118
+0.22%
None 0.9480
0.6209
0.5591
0.3367
0.4825
0.5541
0.1648
1.5611
SNN
+Ours
0.9484
0.6232
0.5547
0.2865
0.4824
0.5551
0.1648
1,5605
+1.81%
None 0.9460
0.6244
0,5505
0.3129
0,4824
0.5521
0.1622
1.5216
DCNv2
+ Ours
0.9454
0.6196
0.5388
0.2629
0.4809
0.5516
0.1623
1.5246
+ +1.64%
None 0.9605
0.6148
0.8342
0.2370
0.4883
0.5550
0.1623
1.4732
TabR
+ Ours 0.9606
0.6233
0.8426
0.2392
0.4827
0.5497
0.1627
1,4620
+0.52%
None 0.9610
0.6341
0.8378
0.2325
0.4804
0.5520
0.1619
1,4857
ModernNCA
+Ours 0.9620
0.6356
0.8316
0.2255
0.4791
0.5535
0.1617
1,4903
+0.30%
None 0.9640
0.6325
0,8290
0.2306
0,4813
0,5500
0.1612
1,4887
TabM
+Ours
0.9629
0.6271
0.8363
0.2321
0.4791
0.5488
0.1609
1.4812
+0.07%
Table 12. Detailed performance results (AUC for classification and RMSE for regression) before and after adopting the proposed temporal embedding method from Section within our training protocol on the TabReD benchmark. These results are plotted in Figure right. All methods demonstrated improvement after using our temporal embedding, with average performance improvement of 0.74%.
C.3. Performance Rankings
We present the performance rankings of the model under different splits, as well as the rankings when temporal embedding is adopted to our training protocol, in Table 13. The results indicate that TabM, CatBoost, and XGBoost consistently perform well, while ModernNCA excels under the random split and our training protocol, and MLP-PLR performs the opposite.
is worth noting that although the relative improvement of TabM over MLP decreases after adding the temporal embedding (-0.12% in Figure 7), TabM itself still achieves 0.07% performance gain (shown in Table 12) and rises by 0.875 ranks in the average model ranking (shown in Table 13). Notably, none of the other methods experience performance drop. This provides comprehensive multi-perspective evaluation.
Splits
MLP
PLR
FT-T
SNN
DCNv2
TabR
MNCA
TabM
XGBoost
CatBoost
LGBM
Original
7.750
4.375
6.875
9.375
8.250
7.375
6.500
3.125
3.375
4.250
4,750
Random
8,250
5.625
5.625
10.250
9.625
8.000
4.750
2.750
3.125
3.125
4.875
Ours
8.000
5.750
7.500
9.500
8.375
8.125
4.875
4.000
3.375
2.125
4.375
Ours temporal embodlling
7.875
6.625
6.250
9.625
9.250
6.250
4.625
3.125
4.500
2.875
5,000
Table 13. Performance rankings of original temporal split in (Rubachev et al., 2025), random split in Figure 2, and our proposed temporal split in Section 4.4 with and without temporal embedding, measured by the average performance ranking the TabReD benchmark, as an extension of Table 1. "PLR," "MNCA," and "LGBM" denote "MLP-PLR:" "ModernNCA." and "LightGBM," respectively.
20

---

Understanding the Limits of Deep Tabular Methods with Temporal Shift
C.4. Model Representations
We present the model representations of TabM before and after incorporating temporal embedding in Figure 10, representative SOTA model complementary to the MLP shown in Figure 5. Since our TabM internally ensembles 32 model representations, compute their average reduce computational cost, which aligns with how the model operates during inference. The left plot exhibits clear grid structure and shows richer representations than MLP, reflecting the stronger representational capacity of TabM. However, the absence of diagonal stripes suggests that the model still primarily captures only two coarse temporal modes (i.e., weekdays and weekends). contrast, the right plot shows the model representation after incorporating temporal embedding, where both the grid structure and diagonal patterns are clearly visible, indicating that the model has learned more fine-gratee iin-grained temporal knowledge.
2.0
5
1.5
4
3
1.0
0.5
Figure MMD heatmaps of model representations of TabM before (left) and after (right) incorporating temporal embedding. The visualizations are obtained by averaging the 32 internal representations of the ensemble. The observed patterns are consistent with those shown in Figure 5.
21


Reviews:
[{"review_id": "M1mCCA0aJ5", "reviewer": "Reviewer_k7kB", "summary": "The paper studies temporal shifts aspects of tabular data\n\nThe paper has two main contributions:\n- First, authors propose a data splitting and validation protocol for improved model performance. The protocol aims to mitigate two phenomena discussed in the paper *1) training lag* (distances between last training timestamp and the first testing timestamp – intuitively the more up-to-date the training data is the better) *2) validation bias* (the difference between `train ↔ val` and `train ↔ test` differences – this can affect model selection in tuing and early stopping)\n- Second, the authors look at the temporal patterns in the data (periodicity, trends) and argue that some architectures fail to capture those. To remedy this authors propose a timestamp embedding, which improves performance for some DL architectures.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback! We will address your concerns in the following responses.\n\n> The authors should be carefull ...\n\nOur experimental setup **strictly follows TabReD**. Moreover, since this is a temporal scenario where each test sample is evaluated individually, the **varying time gaps between each sample and training set** reflect real-world conditions. \n\nAdditionally, **this does not impact our training protocol**, which only provides **guidance stated in lines 307–316**. Extensive experiments have validated the robustness of our training protocol, even when the training lag cannot be reduced to zero (**(c) vs. (d)** in Figure 3).\n\nThe newly proposed temporal split is **primarily intended to illustrate the effectiveness of this protocol**, reinforcing our focus on **analyzing** training lag and validation bias.\n\n> Temporal emb is not fully studied. Hardly helps SOTA.\n\nThis issue is already identified in our paper (**lines 401, 410**). Our temporal emb **converts timestamps into numerical inputs**, which may be **incompatible with PLR emb**. Specifically, once timestamps are embedded, their representation reflects temporal similarity. Applying another periodic transformation via PLR could **increase optimization difficulty**.\n\nDirectly feed the temporal emb into the model backbone **consistently improves**: MLP-PLR +0.01% → +0.26%, TabM +0.07% → +0.15%, MNCA +0.30% → +0.38%. Please refer to Table D in the repository: https://anonymous.4open.science/r/Tabular-Temporal-Shift-BCCA/.\n\n> How does new strategies affect model comparisons?\n\nPlease refer to our response to Reviewer 2MA7.\n\n> Fig 3 does not consider the confound effects ...\n\nOur current experiments **already addressed this issue**. Our analysis of the loss distribution (**Fig 4 right**) isolates the impact of validation and test set variations while fixing the training set.\n\nIn splits (a,c), with the training set fixed earlier in time and shifts in validation and test sets, we observe in **lines 234-251** the impact of reducing training lag and validation bias.\n\n>The Ours variant is not present for the (a,b,c,d) splits of equivalent size in fig 3.\n\nThe (a,b,c,d) splits are used **only for analysis** of training lag, validation bias, and equivalence. These splits were created by **discarding data** due to dataset limitations. In contrast, both the original and our temporal splits use the entire dataset before $T_{train}$.\n\n> Figure 6 done just for MLP.\n\nWe will add visualizations for SOTA model representations.\n\n> The result on HD excluded from all comparisons.\n\n**HD dataset was excluded only from the comparison in Fig 2**. This is because Figure 2 calculates the improvement of other methods relative to MLP, and MLP performs poorly on the HD dataset. As a result, the relative improvement is significantly larger (~80%) compared to the average improvement on other datasets (<6%). Including this dataset in the mean obscures the result. \n\nHowever, all other comparisons do not rely on relative improvements over MLP, **so the HD dataset is included in all other results**. Suggested by Reviewer 2MA7, we will adopt **robust average** to compute the mean improvement in the revision to better handle such cases.\n\n> TabReD suggest that MLPs without numerical embs ...\n\nHere we are not referring to **numerical embs during training**, but **feature encoding during preprocessing**, such as noisy-quantile encoding. This will be clarified in the revision.\n\nThis aspect is not mentioned in the TabReD paper but can be inferred from the code:\n https://github.com/yandex-research/tabred/blob/main/exp/mlp/homecredit-default/tuning.toml#L54, which assigns noisy-quantile and ordinal for the MLP method on the HD dataset. **TabReD assigns different encodings to different method-dataset pairs**.\n\nWe believe this introduces fairness inconsistencies. Special encodings (e.g., noisy-quantile) do not always improve performance. Therefore, we used only basic encoding (**none for numerical and one-hot for categorical features**) for a fair comparison, despite lowering MLP’s performance on the HD dataset. This explains the difference between our results and those in the TabReD paper.\n\n> Did I understand Fig 2 correctly ?\n\nYes. We will further clarify this.\n\n> What is 1D PLR?\n\nSorry for this confusion. Our proposed temporal emb is specifically designed for the timestamp, we choose a **PLR emb applied to the single timestamp input** as a baseline.\n\n> Are SoTA models able to learn ...\n\nWithout timestamp information, models cannot learn the order and periodicity of samples. **The challenge is incorporating temporal information**. Fig 8 shows that treating timestamps as numerical features causes performance drop, aligning with the common practice of removing timestamps. Only by combining temporal emb with periodicity knowledge can temporal patterns be effectively incorporated.\n\nDue to the limit, some responses may lack detail. Please feel free to raise any further concerns!", "claims_and_evidence": "Claims are adequately (albeit not fully) supported by the experimental evidence.\n\nHere are the key aspects that should be expanded upon in the experiements in my view:\n\n**1**) The first contribution (regarding splitting strategies) makes a claim that the proposed non-standard validation procedure is better for each model, mainly relying on the fact that hold-out future test scores are better. \n\nThe authors should be carefull with proposing a new general evaluation procedures (moreover non-standard ones with \"backward in time\" validation) because 1) better test scores do not necessarily indicate that benchmark (including the protocol) is a better reflection of real-world conditions and model performance (for example, the near-zero train/test gap might be impossible in real-world deployments). \n\nBest solution I see for this issue is – instead of proposing a new better protocol, position the work as an analysis of e.g. train/test gap, or validation bias and expanding such analyses.\n\n**2**) In the second contribution introducing the mechanisms and the utility of the periodic time embedding is not fully studied (e.g. its effects are not universal across model types)", "methods_and_evaluation_criteria": "- Comparisons are done model-vise – e.g. all model types individually did improve their test scores, how does new strategies affect our knowledge regarding model comparisons? Does this make results more consistent with prior benchmarks? Plots with relative improvement over the baseline MLP (trained in the same setting as all other methods could be better suited for answering such questions).\n- The evaluation done in figure 3 do not consider the confound effects of varying the training set. Some additional experiments where only the validation and test sets are changed to model train/test gaps and validation bias effects would greatly imporve the robustness of results.\n- In figure 3. The \"ours\" variant is not present for the (a,b,c,d) splits of equivalent size.\n- Figure 6 analysis is done just for the smiplest MLP model. Results from figure 8 suggest that proposed embedding only helps when using simplest (not SoTA models). This effect is not investigated. Why is it so? Are SoTA models able to learn the temporal patterns without any additional embeddings?", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "There are issues in the experimental analyses that need further investigation:\n\n**1**) The result on the Homecredit Default dataset excluded from all comparisons, the exlanation of the sub-par performance provided by authors in the appendix does not seem correct:\n> One key difference in the implementation is that the (HD) HomeCredit-Default dataset suffers from severe class imbalance, which makes it difficult for methods with limited feature extraction capabilities, such as MLP, SNN (Klambauer et al., 2017), and DCNv2 (Wang et al., 2021b), to perform well. TabReD utilizes numerical feature encoding, which may significantly improve the performance of these models on this dataset, but the improvement is not substantial on other datasets. \n\nPublicly available TabReD benchmark results, suggest that MLPs without numerical feature embeddings (I interpreted encoding here as – embeddings for numerical features, please correct me if I'm wrong).\n\nAdditionally, results on the SH (Sberbank Housing) dataset for the MLP are significantly worth here than the ones reported in the TabReD paper.\n\nOtherwise, the protocol is solid, I don't see other problems.", "relation_to_broader_scientific_literature": "The paper extends observation made in the original TabReD benchmark paper and investigates the aspects of non i.i.d validation.\n\nI find the discussion of similar procedures lacking (that are used in financial data analysis, priror kaggle competitons).\n\nExamples of relevant work in this area:\n- \"A survey of cross-validation procedures for model selection\" https://arxiv.org/abs/0907.4728 (Section 8.3)\n- \"CVTT: Cross-Validation Through Time\" https://arxiv.org/abs/2205.05393 (and related work mentioned there)\n- Purged K-Fold cross validation, described in from \"Advances in financial Machine Learning (2018)\" is often used in competitons when \"backtesting\". It includes validation at different time periods.", "essential_references_not_discussed": "No essential references were missed, *but* the above discussion of relation to broader scientific literature points to relevant areas that were not discussed (data splitting strategies with non i.i.d. data in neighboring domains).", "other_strengths_and_weaknesses": "Strengths:\n\n-  A more in-depth look at handling of temporal non-i.i.d tabular data is timely and interesting, the paper is doing important work (I do not want this review and comments to discourage the ovearll direction the work is taking)\n- The analysis and insights into both splitting strategies and other temporal patterns in datasets are new and interesting.\n- The writing is good, but some points could be iterated upon and improved (I had trouble digesting the arguments)", "other_comments_or_suggestions": "Mostly about the presentation and writing:\n- When describing the experiments with training lag and validation bias, I think a clearer way (this was how I understood this better) is not to communicate the results would be e.g. split (a) has less bias compared to split (c) (not that it is without bias).", "questions_for_authors": "- Did I understand correctly that in Figure 2. You split on train and val randomly but leave the test out temporally? If so, this should be better communicated.\n- What is \"one-dimension PLR embedding\"?\n- Are SoTA models able to learn the temporal patterns without any additional embeddings?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hsAcUZviPI", "reviewer": "Reviewer_dxDF", "summary": "The paper tackles the problem of how deep tabular methods deteriorate under temporal distribution shifts, where data distributions evolve over time. It demonstrates that typical temporal splitting (training on earlier data, validating on data just slightly more recent, and then testing on even later data) can hinder performance because of a training lag (lack of recent training examples) and validation bias (the validation split may not fully reflect the larger distribution shift faced at test time). By carefully analyzing these issues, the paper proposes a new temporal splitting protocol that reduces training lag and validation bias and thereby achieves performance closer to that of a random data split, while maintaining temporal realism.\n\n\n## update after rebuttal\nAfter carefully considering the points you presented and other reviewers' comments, I still believe that my initial evaluation remains accurate. Therefore, my score remains unchanged.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful comments! We will address your concerns in the following responses.\n\n> The authors do not seem to provide the code, so I remain conservative about the results reported in the paper.\n\nOur code is now available at https://anonymous.4open.science/r/Tabular-Temporal-Shift-BCCA/. Enjoy the code!\n\n> The temporal emb approach is tested primarily on a fixed set of known cycles. \n\n**The focus of this paper is to analysis why models perform poorly in temporal shift scenarios.** We identified the absence of temporal information in model representations and **improved performance by introducing our temporal embedding, thereby completing our argument**.\n\nHandling unknown or variable cycles will be the focus of **future work**, and we will include this discussion in the **limitations** section. Regarding the approach to learning **unknown or variable cycles**:\n\n- A common approach is to **reweight training samples** [1], but such methods **require an accurate validation set** (instead of a validation set similar to the test time), which is **difficult to obtain in temporal shift tasks**.\n- Another method is to **introduce a matching mechanism** between the training and test set distributions (e.g., attention), but this **requires having a test distribution at test time**, meaning multiple test samples must be obtained, which is closer to test-time adaptation [2].\n\nBased on the MMD visualization and experimental results comparison presented in the paper, we believe the existing fixed cycles already **effectively cover most scenarios (lines 379-384)**. We have also provided experimental results with variable cycles, as addressed in the next question.\n\n> Have you tried letting the model learn different frequencies if the known cycles are not relevant?\n\nYes, we also experimented with **tuning hyperparameters** to find the cycles, results in **Table C in the repository**. When using fixed prior periods, ModernNCA achieved a 0.30% performance improvement, while **setting adjustable cycles resulted in a -2.48% performance decline**.\n\nIn temporal distribution shift scenarios, due to the absence of an entirely accurate validation set, we believe that **prior knowledge of fixed cycles is more stable and interpretable than adjustable cycles**.\n\nIt's also important to note that in many tasks, **complete cycles are not available**. For example, in the weather dataset, there is a yearly cycle, but the training set does not span a full year, which highlights the importance of prior knowledge.\n\n> Are there practical guidelines for selecting these cycles or discovering new cycles automatically in domain-specific tasks?\n\nFor domain-specific tasks, we still recommend **using prior cycles informed by expert knowledge** or setting them **based on MMD visualization**.\n\n> Is it possible to cause data leakage problems?\n\nThis splitting method does not introduce data leakage, as no information that would be unavailable at deployment is used during training. Therefore, **the model's performance on the test set remains reliable**.\n\nWe hope this response addresses your concern. Please feel free to raise any further questions!\n\n---\n\n[1] Mengye Ren et al. Learning to Reweight Examples for Robust Deep Learning. ICML 2018: 4331-4340\n\n[2] Changhun Kim et al. AdapTable: Test-Time Adaptation for Tabular Data via Shift-Aware Uncertainty Calibrator and Label Distribution Handler. CoRR abs/2407.10784", "claims_and_evidence": "Yes. The claims made in the paper are mostly empirically validated with comprehensive experimental results.", "methods_and_evaluation_criteria": "Yes. The temporal shift problem is common and the proposed method seem to resolve it accordingly as expected.", "theoretical_claims": "The paper does not seem to make theoretical claims.", "experimental_designs_or_analyses": "The experiments appear thorough and carefully controlled, with clear metrics and repeated random seeds. The design supports the main conclusions well.\n\n1. **[Important] Experimental Design:** The authors carefully **ablate different splitting strategies** in terms of training/validation intervals, time-lags, reversed splits, etc. This isolates the roles of lag, bias, and “equivalence” in validating the test distribution.\n2. **Qualitative Analysis:** They use **MMD heatmaps** to illustrate how the model’s learned feature distributions differ across time, giving a qualitative sense of whether or not periodic/temporal patterns are preserved.", "relation_to_broader_scientific_literature": "The paper also aligns with broader trends in studying temporal distribution shifts in fields like time-series forecasting and handling “open-environment” learning.", "essential_references_not_discussed": "To the best of my knowledge, the paper should have included the essential references.", "other_strengths_and_weaknesses": "**Strengths**\n\n1. **Insightful diagnosis** of where conventional temporal splits fail: the paper clarifies the subtle difference between a time-based approach that is correct for causal or forecasting tasks vs. real-world tabular tasks that remain partly cross-sectional.\n2. **Lightweight method** (Fourier embedding) that is easy to replicate in practice, plus a well-reasoned splitting procedure.\n3. **Extensive experimental evidence** with thorough ablations, multi-seed runs, and MMD-based visualizations.\n\n**Weaknesses**\n\n1. **[Important]** The **temporal embedding** approach is tested primarily on a fixed set of known cycles (daily, weekly, monthly, yearly). Real data might have domain-specific cycles or more complex patterns that could require further tuning.\n2. The authors do not seem to provide the code, so I remain conservative about the results reported in the paper.", "other_comments_or_suggestions": "1. It might help if the authors analyzed the cost of ignoring certain periods or the potential mismatch between a fixed Fourier period (e.g., 7 days) and real data that might have a different cycle.", "questions_for_authors": "1. **[Important]** The authors focus on daily, weekly, monthly, and yearly cycles. Are there practical guidelines for selecting these cycles or discovering new cycles automatically in domain-specific tasks? Have you tried letting the model learn different frequencies if the known cycles are not relevant?\n2. **[Important]** As the proposed splitting method do not seem to keep the original temporal order of the data samples, is it possible to cause data leakage problem? For instance, if an older sample is used for validation, and the corresponding newer sample is used in training.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "TU1MAj5oT3", "reviewer": "Reviewer_2MA7", "summary": "The paper investigates the impact of temporal shift in tabular data and presents a set of solutions to mitigate its effects. Since tabular data instances are typically collected in chronological order, temporal shift naturally arises. The authors first find that the commonly used time-based validation split results in worse performance compared to random splitting, and propose a refined temporal splitting protocol designed to reduce training lag and verification bias. Then, from the perspective of model representation, the authors find that the existing methods fail to capture temporal information, and propose a temporal embedding method for deep methods. The experimental results show that both the splitting protocol and the temporal embedding significantly improve the performance of the model under temporal shift scenarios.\n## update after rebuttal: Most of my concerns are addressed", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your constructive suggestions! We will address your concerns in the following responses.\n\n> The detailed design for these different methods is not discussed.\n\nWe apologize for this oversight. We will add an additional section in the preliminary in the revision to introduce the fundamentals of learning from tabular data.\n\n> They present the percentage change in the robustness score, which may not be intuitive.\n\nThank you for your suggestion! We have now additionally provided the **change in standard deviation** in **Table B** in the repository: https://anonymous.4open.science/r/Tabular-Temporal-Shift-BCCA/. In our comparison, while our method results in a slightly higher standard deviation than the original split (+16.7%), it achieves a significantly lower standard deviation compared to random splitting (+154%). \n\n> It is recommended to use a metric that is robust to outliers.\n\nWe agree. In the revision, we will replace the average percentage change calculation with a **robust average**, which excludes the maximum and minimum values when computing the percentage change across the eight datasets. We have updated Figure 2 by **Figure A** in repository, and **it continues to support the same conclusion**: random splitting significantly improves model performance compared to the original split and aligns more closely with existing benchmark results.\n\n> Include a performance comparison of the methods under different protocols.\n\nWe have included a **comparison of model performance under different protocols**, specifically after applying our splitting method and temporal embedding, shown in **Figure A bottom**. All comparisons use the **robust average** to compute the average percentage change relative to the original split for MLP, ensuring direct comparison. Additionally, we provide the **average rank** of the models for a more comprehensive evaluation, shown in **Table A**.\n\nWe hope this response addresses your concern. Please feel free to raise any further questions!", "claims_and_evidence": "Yes, the claims are generally well-supported by experimental results.", "methods_and_evaluation_criteria": "Yes, the proposed splitting protocol and temporal embedding are all tested on the TabReD benchmark, which focuses on temporal shifts in tabular data.", "theoretical_claims": "This paper focuses on experiment design and analysis and does not make any theoretical claims.", "experimental_designs_or_analyses": "The experimental design is well-structured and effectively evaluates the proposed claims and methods.\n\nSplitting protocol: Four new splits are used to verify the effects of reducing training lag, reducing validation bias, and ensuring the equivalence of the validation set, respectively, with controlled variables. The use of MMD visualization and loss distribution further supports these claims.\n\nTemporal embedding: The authors first identify the missing temporal information in MLP representations, including multiple periods and trend information. They then propose a temporal embedding that introduces period and trend information into the model. The only potential issue is that the authors do not present the model representation after incorporating their temporal embedding.", "relation_to_broader_scientific_literature": "Tabular learning is generally based on the i.i.d. assumption, but this paper focuses on temporal shift, which has strong practical significance. Among recent deep tabular models, retrieval-based methods like ModernNCA have demonstrated excellent performance but are considered to perform poorly in distribution shift scenarios. This paper shows that by applying a refined splitting protocol and temporal encoding, retrieval-based methods can regain competitiveness.", "essential_references_not_discussed": "This paper sufficiently reviews the relevant literature.", "other_strengths_and_weaknesses": "Strengths:\nThe paper is well structured and easy to follow.\nThe experiment designs and results are convincing.\n\nWeaknesses:\nThe paper compares various types of models (e.g., retrieval-based methods, ensemble-based methods) and provides some analysis in the experimental section, such as “the importance of no-lag candidates for retrieval-based methods.” However, the detailed design for these different methods is not discussed in the related work or preliminary sections.\n\nThe authors claim that their splitting protocol achieves similar performance to random splitting while improving stability. However, they present the percentage change in the robustness score, which may not be intuitive.\n\nThe performance of the MLP on the HD dataset is significantly weaker than that of the other methods, and the authors excluded this dataset from the percentage improvement calculation for the other methods on the MLP. This could potentially cause confusion. It is recommended to use a metric that is robust to outliers.", "other_comments_or_suggestions": "There are some typos in the paper. The authors only show the performance improvement of the model after applying the splitting protocol and temporal embedding. It is recommended to include a performance comparison of the methods under different protocols.", "questions_for_authors": "Please refer to the weaknesses and comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1B8WMqSDU0", "reviewer": "Reviewer_nBxA", "summary": "- The paper analyses temporal splits for tabular DL. It proposes a new splitting strategy and also analyzes how random splitting affects performance. Additionally, the authors propose temporal embeddings, using fourier transformation and somewhat following the ideas proposed in [1] with PLR embeddings\n\n\n---\n[1] Gorishniy, Yury, Ivan Rubachev, and Artem Babenko. \"On embeddings for numerical features in tabular deep learning.\" Advances in Neural Information Processing Systems 35 (2022): 24991-25004.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback! We will address your concerns in the following responses.\n\nFirst, we would like to clarify several key differences between our work and TabReD and PLR.\n\n1. **Difference from TabReD**: TabReD shows that real-world tabular datasets, inherently containing temporal shifts, require **temporal splits for realistic test sets**. Random splits may lead to misleading assessments, and TabReD demonstrates that temporal splits significantly alter model rankings. Our work builds upon TabReD by identifying **validation set splitting** as a crucial factor affecting model performance once the test set position is fixed. We find that, in this setup, randomly splitting the validation set yields better results than using the temporal split in TabReD. We further analyze the underlying reasons behind this observation and propose our refined temporal split. Therefore, our contribution focuses on **validation set splitting**, whereas TabReD primarily addresses **test set splitting**, making the two approaches complementary rather than overlapping.\n2. **Difference from PLR emb**: Our temporal emb differs from PLR emb both in **scope and design**. PLR emb is a numerical feature emb method that samples periodicities from $N(0, \\sigma)$ to capture cycles in numerical features. In contrast, our temporal emb is specifically designed to incorporate **timestamp information** into models, then treat it as a numerical input feature. It is a **plug-and-play** approach. Structurally, our method relies on **prior cycles**, making it more suitable for handling temporal patterns. Additionally, we explicitly address the challenges of **multi-period coupling** and **trend representation**, which are essential in temporal settings but are not considered in PLR emb. We also **consider PLR emb a baseline** when designing our temporal emb (figure 8 left).\n\nWe hope this clarifies our contributions. Our code is available at https://anonymous.4open.science/r/Tabular-Temporal-Shift-BCCA/, with additional results.\n\n> Random split perform identical.\n\n**The random split serves as a baseline**. By analyzing its differences with temporal splits in TabReD, we identified the impact of training lag and validation bias, **motivating our temporal split strategy**.\n\n**While the random split performs well, it suffers from instability** (Std increased by 154%). Our proposed temporal split not only maintains competitive performance but also significantly improves stability, as shown in **Table B in repository**.\n\n> How are non DL/autoregressive/ICL methods effected by the splits?\n\n1. We have presented the impact of non-DL methods (including Linear, Boosting methods, and Random Forest) under different splits in **Figure 2, Figure 5, and the Appendix (page 13)**. These methods consistently show improvements, with generally better performance in our new temporal split.\n2. We tested the **Mambular** method. Due to the large size of the TabReD dataset and the inefficiency of autoregressive methods, we only provided results for six datasets. Under our split, its performance showed a significant improvement (+4.56%), as shown in Table E in repository.\n3. We tested **TabPFN v2**. Since the general model does not require training, we adjusted its context samples: in the Original split, we randomly selected 10,000 context samples, while in Ours, **we selected the 10,000 context samples closest to the test set**. This also led to an improvement (+0.71%), as shown in Table E.\n\n> Since Fig 8 you analyze random splits, how does random split compare in Fig 3?\n\nFigure 8 focuses on comparing **our updated temporal split** with the improvement brought by adding temporal embs. **The random split is only presented in Figures 2 and 5**.\n\nIn Figure 3, the bar chart on the right compares the performance of four specifically constructed splits to analyze the effects of reducing training lag, mitigating validation bias, and ensuring validation set equivalence. **Since these splits contain different amounts of data, their results cannot be directly compared to those of the Original, Ours, or Random splits**.\n\n> Where are the splits a), b), c) and d) coming from?\n\nThe splits a), b), c), and d) are **entirely our contribution**. Our work focuses on different aspects compared to TabReD.\n\n> Why is \"Ours\" not shown in the graphic on the right?\n\nThe (a,b,c,d) splits are used **only for analysis** of training lag, validation bias, and validation equivalence, which can be seen as an **ablation study**. These splits were created by **discarding data** due to dataset limitations. In contrast, both the original and our temporal splits use the entire dataset before $T_{train}$. As a result, **they cannot be directly compared with the Original or Ours**.\n\nWe hope this response addresses your concern. Please feel free to raise any further questions!", "claims_and_evidence": "-  The claim that the proposed new temporal split is superior is not supported by the results (Figure 5) as the random split seems to perform nearly identical\n   - While that claim is not met, the finding that the random split outperforms the split from Rubachev (2025) is an interesting finding itself. If the claims in the abstract/contributions that the newly proposed method \"offers substantial improvements\" are scaled down a bit, this is not an issue\n- The proposed temporal embedding does also not offer any improvement over the known PLR embeddings (Figure 8)", "methods_and_evaluation_criteria": "- Yes the benchmarks are very solid, however there are some questions I have regarging the results:\n    - Figure 8 on the left you have MLP and MLP-PLR but then have PLR as an embedding strategy. What do you use in MLP-PLR for the other embedding strategies and how does MLP differ from MLP-PLR when using PLR embedding strategy?\n    - Since in Figure 8 you analyze the random splits, how does the random split compare in Figure 3?", "theoretical_claims": "- Yes, however there are no proofs/theoretical contributions", "experimental_designs_or_analyses": "- The experimental design seems very solid with the very new and interesting models being analyzed. \n    - However, I would be interested in how truly autoregressive tabular models are affected by the splits and by your embeddings [1].\n    - How are non DL models effected by the splits? I.e. boosting models, or PFNs?\n- More importantly, no code is provided during submission. While all results and used benchmarks are very consistent, reproducible code should be provided already during submission time.\n\n---\n[1] Thielmann, Anton Frederik, et al. \"Mambular: A sequential model for tabular deep learning.\" arXiv preprint arXiv:2408.06291 (2024).", "relation_to_broader_scientific_literature": "The paper very directly relates to the TabRed work proposed by [1] as also noted by the authors. Additionally, the temporal embeddings are also minor adjustments to the work proposed by [2]. The -in my opinion- most interesting finding, that random splits seem to work very well is not adequately analyzed and addressed as it directly confronts the ideas presented in [1].\n\n---\n[1] Rubachev, Ivan, et al. \"TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks.\" arXiv preprint arXiv:2406.19380 (2024).                           \n[2] Gorishniy, Yury, Ivan Rubachev, and Artem Babenko. \"On embeddings for numerical features in tabular deep learning.\" Advances in Neural Information Processing Systems 35 (2022): 24991-25004.", "essential_references_not_discussed": "- I wonder how these splits effect ICL models (TabICL, TabPFN v2) [1, 2]. \n- Other than that all necessary work is included and adequately adressed.\n\n\n---\n[1] Qu, Jingang, et al. \"TabICL: A Tabular Foundation Model for In-Context Learning on Large Data.\" arXiv preprint arXiv:2502.05564 (2025).\n[2] Hollmann, Noah, et al. \"Accurate predictions on small data with a tabular foundation model.\" Nature 637.8045 (2025): 319-326.", "other_strengths_and_weaknesses": "- The paper is overall very well written and the benchmarks/tests seem extremely solid. Models as new as TabM (ICLR 2025) are already included.", "other_comments_or_suggestions": "typos:\nAbstract, line 028 \"analyses\" should be \"analyze\"\n\nintroduction:\nNot just continuous/categorical for regression classification, i.e.distributional approaches, count data/poisson", "questions_for_authors": "-  Where are the splits a), b), c) and d) coming from? Is that from Rubachev (2025) or your contribution? and why is \"Ours\" not shown in the graphic on the right?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper explores the impact of temporal shifts in tabular data and identifies two critical factors—training lag and validation bias—that hinder generalization. To address these, the authors propose a refined data-splitting protocol and introduce a lightweight temporal embedding module that enhances temporal representation learning. These two contributions lead to consistent performance improvements across a variety of tabular methods on the TabReD benchmark.

The paper is well-written and addresses a practically important yet under-explored challenge. Its strengths include novel insights into common evaluation pitfalls, broad empirical validation across diverse models and datasets, and a temporal embedding that is simple, effective, and easily integrable with existing tabular architectures—especially benefiting non-temporal models.

Some reviewers raised concerns regarding the generalizability of the proposed splitting strategy, the relatively smaller gains of the temporal embedding on more complex models, and its reliance on prior knowledge of temporal patterns. The authors provided a comprehensive rebuttal, effectively addressing most of these concerns. While the proposed splitting strategy may not be universally applicable, it draws valuable attention to the importance of evaluation protocol design in temporally shifting environments and offers a practical baseline for future work. Following the discussion, reviewer consensus improved, resulting in final scores of 3, 3, 3, and 4.

Overall, the paper makes a meaningful and empirically grounded contribution to deep tabular learning under temporal shift. I recommend acceptance.
Decision: Accept (poster)