Temporal Generalization: A Reality Check
Authors: Divyam Madaan
Sumit Chopra
Kyunghyun Cho
Venue: ICLR 2026 Poster (2026)
Keywords: Temporal Generalization and Extrapolation
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: The key to temporal generalization is not to design new algorithms, but to identify the reasonable assumptions about how the data generating process evolves over time.

Abstract: Machine learning (ML) models often struggle to maintain performance under distribution shifts, leading to inaccurate predictions on unseen future data. In this work, we investigate whether and under what conditions models can achieve such a generalization when relying solely on past data. We explore two primary approaches: convex combinations of past model parameters (parameter interpolation) and explicit extrapolation beyond the convex hull of past parameters (parameter extrapolation). We benchmark several methods within these categories on a diverse set of temporal tasks, including language modeling, news summarization, news tag prediction, academic paper categorization, satellite image-based land use classification over time, and historical yearbook photo gender prediction. Our empirical findings show that none of the evaluated methods consistently outperforms the simple baseline of using the latest available model parameters in all scenarios. In the absence of access to future data or robust assumptions about the underlying data-generating process, these results underscore the inherent difficulties of generalizing and extrapolating to future data and warrant caution when evaluating claims of such generalization.

Published as a conference paper at ICLR 2026
TEMPORAL GENERALIZATION: A REALITY CHECK
Divyam M Sumit Chopra Kyunghyun Cho Courant Institute Mathematical Sciences, New York University! Center for Data Science, New York University² Grossman School of Medicine, New York University
CIFAR LMB
[divyes madaan, sumit, kyunghyun. cho
ABSTRACT
Machine learning (ML) models often struggle to maintain performance under dis- tribution shifts, leading to inaceurate predictions unseen future data. In this work, we investigate whether and under what conditions models can achieve such generalization when relying solely past data. We explore two primary ap- proaches: convex combinations of past model parameters (parameter interpola- tion) and explicit extrapolation beyond the convex hull of past parameters (param- eter extrapolation). We benchmark several methods within these categories on a diverse set of temporal tasks, including language modeling. news summarization, news tag prediction, academic paper categorization, satellite image-based land use classification over time, and historical yearbook photo gender prediction. Our em- pirical findings show that none of the evaluated methods consistently outperforms the simple baseline of using the latest available model parameters in scemarios. In the absence of access to future data robust assumptions about the underly- ing data- eetta-gnnrraing process, these results underscore the inherent difficulties generalizing and extrapolating to future data and warrant caution when evaluating claims of such generalization.
INTRODUCTION
"Prediction very difficult, especially about the future." " Niels Bohr
Temporal generalization of machine learning (ML) models is challenging, but critically important. After being trained on retrospective data, these models are deployed in real-world applications, par- ticularly in high-stake domains such as finance, healthcare, and autonomous systems, where the distribution the data the model sees could drift over time. Failures in these contexts can lead to severe financial losses or pose significant risks human safety. It is therefore crucial to develop and evaluate models under deployment scenarios that accurately reflect temporal evolution. Yet, despite significant progress and widespread adoption, ML models suffer from performance degradation on data collected after the training period (Jaidka et al., 2018; Luu et al., 2021; Nylund et al., 2024). This issue of temporal performance degradation widespread, affecting even large-scale state-of- the-art models such as GPT-4 (Achiam et al., 2023) and Gemini (Team et al., 2023). Figure 1 (left) shows this challenge, illustrating a 30% increase in perplexity when a model trained on past news summarization data is evaluated on future months, compared to model adapted with more recent data. Such degradation in temporal generalization can lead to real-world consequences, The sub- stantial financial and computational costs associated with training these foundation models (Cottier et al., 2024) make frequent retraining economically impractical. This necessitates alternative strate- gies for maintaining model utility over time.
With complete knowledge of the underlying data mataa-enerating process, it would, in principle, be pos- sible to construct a model that generalizes effectively future data. However, in most practical settings, this assumption is overly idealized and rarely holds. Motivated by this limitation, this paper addresses the following central question:
Can we build a model generalizable to the future without any access to future data and imperfect knowledge of the data- gena-agnn process?

---

Published as a conference paper at ICLR 2026
Cumulative Avg Perploxty
1.7
0
fa.
1.6
1.5
81
f0
1.4
-
12
fn
8
11
2012:03
2012 09
2013-83
2013
2013-09
2014-03
20144-09
2015
2015.0
2016-03
2018-00
D1 D2
Training
Evaluation
Time
Figure I: (Left) Performance degrades over time. The widening performance gap between a stale model trained once in January 2012 (red) and monthly updated model (green) illustrates the decay in performance over time. The evaluation was conducted using data from March 2012 onward. (Right) Temporal generalization framework. Across sequential learning stages (S, the y- axis), a model fe, is trained on data Dr (orange) initialized with 0,-1 from the previous stage. This generates a sequence of historical parameters. This sequence is used to estimate future parameters 0.+8, which are used to evaluate future data (blue) D for 6 (x-axis).
We adopt stringent yet realistic setting to evaluate temporal generalization, as illustrated in Fig- ure (right). At given time t, we have access to a sequence model parameters (or checkpoints) which resulted from training on historical data. Our objective is to leverage this se-
0.} (O,....0;, quence of checkpoints estimate a new set of parameters, denoted 0.+8. that will exhibit temporal generalization on unseen data D:+8 at a specific time into the future t + (where ò >> 0).
In addressing this question, posit that any approach that only leverages retrospective data can be grouped into two categories: one that interpolates based on historical parameters and another that explicitly extrapolates towards the future.
Parameter interpolation (Ilharco et al., 2022; Yadav et al., 2024; Davari and Belilovsky, 2024; Jang et al., 2024; Dziadzio al., 2024). This conservative yet intuitive approach restricts the search space for new parameters to the convex hull defined by past parameter checkpoints. We explore model merging (Ilharco al., 2022; Yadav al., 2024; Davari and Belilovsky, 2024) and a sim- ple downscaling of the recent model parameters. The underlying intuition is twofold: parameters from the recent past might contain information relevant to the near future, and simple scaling might mitigate model overconfidence unseen future data.
Parameter extrapolation (Nasery et al., 2021; Bai et al., 2023; Cai et al., 2024; Nylund et al., 2024). A more ambitious approach involves explicitly estimating future model parameters by ex- trapolating beyond the convex hull defined by the trajectory of past parameter values. This method operates under the hypothesis that the temporal evolution of model parameters, observable from historical training dynamics (Nylund et al., 2024), can inform predictions about future parameters, even in the absence of future data access. In particular, we investigate whether a Taylor series-based approximation can facilitate effective extrapolation of model parameters into the future.
We conducted comprehensive empirical study comparing several methods representative of these two categories. Our evaluation spans a diverse array of temporal datasets and tasks, such as lan- guage modeling on evolving news corpora, news summarization and tag prediction, categorization of academic papers, land use classification from satellite imagery reflecting changes over several years, and gender prediction from historical yearbook photos spanning multiple decades.
There are significant challenges that complicate the rigorous evaluation of temporal generalization methods. Existing studies are limited to simplistic setups and lack real-world applicability due to high computational cost (Nasery et al., 2021; Bai et al., 2023; Cai et al., 2024), non-transparent
hyperparameter selection (Bai et al., 2023; Cai et al., 2024; Nylund et al., 2024) and the use of future data during model adaptation (Nylund et al., 2024; Cha and Cho, 2024). Additionally, the inherent non ioniientifiabilitty of deep neural networks (Hecht-Nielsen, 1990; Sussmann, 1992; Phuong and Lampert, 2020) yields non-linear parameter trajectories, which complicates both interpolation and extrapolation. Addressing these prevailing gaps, we provide the first holistic evaluation of these interpolation and extrapolation methods within large-scale settings.
2

---

Published as a conference paper at ICLR 2026
The findings of this study highlight the intrinsic challenges of forecasting future model parameters based solely on historical trajectories, particularly in the absence of access to future data distributions or validated assumptions about the underlying data-gen generating process. Among the evaluated meth- ods, the only one that consistently avoided performance degradation across datasets was the down- scaling of recent model parameters. To better understand these outcomes, we conduct detailed tem- poral analysis of parameter dynamics and discuss the implications of key design choices. We hope these insights will inform future research on the development and evaluation of temporally robust machine learning models. The code is available at https: ////tttttttt github. com/ divyam3899 /TG.
Contributions. This work investigates temporal generalization through a large-scale evaluation of parameter interpolation and extrapolation. We adhere to the strict constraint of no access to the future and evaluate with multiple monthly and yearly temporal datasets and model architectures. Our principal empirical finding is that these parameter interpolation and extrapolation methods fail to improve, and often degrade, performance compared to using the most recent model. Our findings underscore the profound difficulty of predicting future model parameters solely from historical data under these stringent settings. This is because, without strong assumptions about how the data- generating process evolves over time or access to the future, the future can be arbitrarily different.
Outline. We begin by formalizing the problem of temporal generalization ($2). Next, we explore multiple approaches for parameter interpolation and extrapolation ($3), describing their underlying assumptions and associated difficulties ($4). Our experimental evaluation ($5) reveals that none of the methods investigated reliably achieve temporal generalization. Throughout analysis, we identify key design principles and highlight directions for future research ($6).
THE PROBLEM OF TEMPORAL GENERALIZATION
Temporal generalization (or temporal domain generalization) (Nylund et al., 2024; Roth et al., 2024; Dai et al., 2024) refers to the ability of MI. models to maintain performance over time. It can be viewed as a specific instance of classical domain generalization (DG) (Sun and Saenko, 2016;
Arjovsky et al., 2019; Yue et al., 2019; Sagawa et al., 2020; Zhou et al., 2020; Zhang et al., 2021; Zhou et al., 2021; Yao et al., 2022b), where each timestamp corresponds to an ordered domain. The objective of temporal generalization is to generalize to future unseen timestamps. In this work, we focus on temporal generalization at a large scale, using models from 70M to 770M parameters on tasks like language modeling, news summarization, and classification without access the future.
Consider a sequence of T datasets, (De) where each D. { =( consists of N₂ data points collected at time t. At any current time f, we obtain model parameters 0g by training on the current data segment D.. We assume access the sequence of past model checkpoints ( 0z)- O, represents the latest checkpoint trained on the recent data D.. Our objective at time 1 is to use historical checkpoints (fI...., 0e) to estimate new parameters 0 that will perform well on unseen future data D.45 from a specific future time + (where 8 > 0).
We focus on the streaming setting, which reflects real-world deployment scenarios. Often retraining on all historical data is computationally infeasible, and models must be continuously updated and evaluated. The estimation of lets occurs without any access to future or past data (see Figure (right)). This constraint of future access provides a stringent and realistic test of a model's ability to stand the test of time using only the past. Ideally, one could capture temporal dynamics by training a
Infeasibility of non-linear approaches. sequence model, such as a RNN on a series of checkpoints (Bai et al., 2023) or by using compressed representations obtained by autoencoder (Cai et al., 2024). These approaches are impractical for large-scale models for two reasons. First, their computational complexity scales linearly with the number of parameters. It is intractable to predict a single large output layer for an entire language vocabulary. Second, they require fine-grained temporal data. Accurately modeling a non-linear parameter-time trajectory in a high-dimensional space a large model requires a density of samples that can grow exponentially. This requirement is unmet by practical constraints, as publicly available datasets are typically too coarse (e.g., yearly snapshots with ten samples for ten-year data). Such parameter prediction techniques have been confined to small-scale models such as a 10-layer LSTM on toy datasets, inapplicable to widely used T5 (Raffel et al., 2020) and densenet-122 121 (Huang et al., 2017) architectures. We provide more detailed discussion of related work in Appendix B.
3

---

Published as a conference paper at ICLR 2026
Time Vectors (Nylund et al., 2024) is the only work that extrapolates language models by decompos- ing parameters into a time-varying language model component 6.-M and a fixed task-specific compo- nent makk While effective in some contexts, this approach has two major practical limitations. First, its success depends on a large, fixed pre-trained model to constrain the temporal updates, which is not applicable to settings without large-scale pre-training, such the Wilds-Time datasets (Yao et al., 2022a). Second, it assumes access to unlabeled future data 0 and supervised future valida- tion data tune hyperparameters; a requirement often unmet in realistic scenarios. In the following section, we focus on methods that are scalable to large models, require no access to future data, and do not assume task-specific parameters to remain constant.
3 PARAMETER INTERPOLATION AND EXTRAPOLATION
In this section, detail two approaches that leverage the past sequence of parameter checkpoints for temporal generalization: parameter interpolation and parameter extrapolation.
3.1 PARAMETER INTERPOLATION
Parameter interpolation constructs future parameters 08+5 as weighted combination of parameters from historical model checkpoints.
Model merging. The general form of parameter interpolation involves creating an explicit weighted average of parameters from multiple past checkpoints. This approach, often referred to as model merging (Ilharco et al., 2022; Yadav et al., 2024; Davari and Belilovsky, 2024; Jang et al., 2024) consolidates diverse historical information to potentially improve generalization to future data Dc+5 and mitigate over-reliance the most recent checkpoint. Formally, the interpolated parameters are a convex combination of the past parameters:
0
O, where 04 2 0 and
02 11 1=0
t=0
Here 0, for i 2 represents the model parameters trained on D;, and 0.- is defined as the zero vector for notational convenience. The hyperparameters a, (for i 2 1) are the weights determining the contribution of each checkpoint. These weights may be uniform (leading to simple averaging) or exponentially decaying (EMA) to assign greater importance to more recent checkpoints. Typically, og is set to zero, limiting the merging to the convex hull of 0.).
While model merging (Ilharco et al., 2022; Yadav et al., 2024; Davari and Belilovsky, 2024; Jang et al., 2024) has shown promise when models are trained on similar tasks or datasets, merging models trained on distinctly different datasets (as can be the case with temporal data) is known to be challenging. This difficulty arises from high loss barriers between the parameter sets (Yamada et al., 2025). Consequently, existing merging or model editing techniques (Ilharco et al., 2023; Yadav et al., 2024; Sagawa et al., 2020; Wang et al., 2024; Fang et al., 2025) are not dircctly applicable to our strict problem setting, as they frequently rely access to data from the target (i.e., future) distribution for model selection and validation. Dziadzio et al. (2024) recently showed that model averaging outperforms complex merging techniques (Yadav et al., 2024) over time. Their evaluation was restricted to curated datasets that did not exhibit strong temporal shifts and contained relatively similar data distributions over time. We revisit model averaging under naturally occurring temporal shifts and show that its effectiveness often diminishes as we evaluate further into the future.
The recent model. A straightforward baseline, and a specific instance of merging framework in Equation (1) is to deploy the most recent model 6, trained on the current dataset D,. This corre- sponds to setting o; = and a; = for i < in the general framework. Since O reflects the latest known data distribution, it is potentially relevant for near-future datasets. Nonetheless, previ- ous studies (Luu et al., 2021; Lazaridou et al., 2021; Zhu et al., 2025; Dai et al., 2024) show that relying on recency can fail to generalize over time with some models collapsing to nearly random performance beyond their training cutoff. This stems from the fact that future data can change ar- bitrarily. In contrast to merely observing this degradation, evaluate whether we can generalize better to the future. We observe that 6,, despite its simplicity, is often surprisingly competitive in time-evolving real-world settings.
investigate proactive strategies to
4

---

Published as a conference paper at ICLR 2026
Parameter downscaling. Another simplification of model merging is focus only the most recent model fe. adjusting its overall magnitude by interpolating it towards the origin 0g = O. Given the parameters Oz, we derive parameters 0245 using a single scaling hyperparameter a::
Oc+5 = a with E [0, 1].
(2)
This is specific instance of the merging framework, where is the weight for Pz. 1 1 for 8g and aj = O for < j < t. Downscaling reduces the parameter norm ||0r] (if a < 1) while preserving parameters optimized for the distribution at time f. This is inspired by empirical and theoretical observations, where parameter norms increase during training (Li and Arora, 2019; and Telgarsky, 2020; Merrill et al., 2021; Nikishin et al., 2022; Dohare et al., 2024; Lewandowski et al., 2025), and larger norms correlate with sharper minima and reduced generalization (Foret al., 2024; Zhao et al., 2022: Yashwanth et al., 2024). A larger norm reflects the model's 's strong reliance on the current data; for temporal generalization, reducing this norm might mitigate overfitting to the present time step. We show the existence of this phenomenon in Figure 4.
its direction (if > 0). The motivation is to prevent over-reliance
3.2 PARAMETER EXTRAPOLATION
more direct, albeit ambitious. approach is estimate the predictive distribution f Bc-s) for any future input Tr+5, where 8 > 0 represents the time increment into the future. To approxi- mate ( 0ets), we consider how the parameters (t') might evolve differentiable function of time l'. Future parameters 0.+8 can be related to current parameters 0g via the approximation lc+s R 02 + 8 0y where 0r = cd0(t') /dd'')/) is the instantaneous rate of change of parameters at time t. The function f could then be approximated via a Taylor expansion at 0,:
[11448 0e+8)= R f(*++8,0) 0.) + 8. (Vof( 0z) 02) + Higher-order terms.
(T 02) is the gradient of f with respect to its parameters 0, evaluated at O, for input Tx+-6- Nasery al. (2021) conceptually similar to this formulation, but also requires specialized mod- els that take t as input. This is incompatible with standard pre-trained models. Computing Equa- tion (3) challenging because it requires computing Vof Oz) for every new future input 21+66 which can be computationally prohibitive. To address this, we extrapolate the model parameters to estimate 0.45- Once 02+5 is obtained, it can be used make predictions f 0e+5). We formalize the parameter-tinn time relationship locally using a Taylor approximation for A(t') centered the current time l. Since direct computation of 0f infeasible without access the underlying generative process of parameters, we approximate it using the most recent checkpoints 0g and 0r Ar:
8e+5 = 0g + a -8 HHigher-order terms R 02 + (0, 01-Ae)
(4) where At > is the time interval between the past parameter checkpoints O₂ and 0:- The scalar a distinct from the future time horizon 6. It defines the extrapolation step size which determines how far along the estimated direction of change (0.=0tat)) we extrapolate. For At example, if A/ represents one discrete time unit of past observations, setting a = AL would cor- respond to a linear extrapolation by an amount of change observed over one such past interval. A smaller positive a (e.g., < a < At) yields a more conservative update along this trend.
AL
Choosing is critical for parameter extrapolation, which we discuss in the following section. While parameters appear to follow smooth trajectories in a low-dimensional space, their evolution in the high-dimensional space can be substantially more complex. We empirically show in Figure 5 that the optimal a often deviates significantly from our assumptions, sometimes even taking negative values, which suggests that interpolative approaches more useful.
4 CHALLENGES WITH TEMPORAL GENERALIZATION
The relationship between model parameters and the function f(. is complex due to the non- convex loss landscapes, which in turn results in issues related to identifiability. This poses challenges for the parameter interpolation and extrapolation methods presented Section 3. This section out- lines these challenges and describes our strategy to mitigate their impact temporal generalization.
5

---

Published as a conference paper at ICLR 2026
4.1 NON-CONVEXITY IN LOss SURFACE RESULTING IN MON-IDENTIFIABIITYY
When models are trained independently at different time steps I, the resulting parameters 0, may reside in disparate basins of the loss surface. Parameter interpolation or extrapolation implicitly as- sumes that parameters from different time steps are connected and lie in the same basin smoothly connected regions. Deep networks rarely satisfy this assumption. There often exist high barriers between different solutions, exacerbated by the lack of identifiability; different parameters O(A) and g(B) can produce the same functionally equivalent input-output mappings f(-, (()) f(..(R)) Retraining yields different local minima due to factors such as weight permutations, initialization, and other random choices, leading to parameter sets that, while functionally similar, converge to different local minima (Hecht-Nielsen, 1990; Sussmann, 1992; Phuong and Lampert, 2020; Chen et al., 2024).
This presents a challenge for parameter analysis over time. For example, if the past parameters from Section 3.1 are distant from each other, interpolation and extrapolation between them might traverse regions of high loss, yielding poorly performing models. The finite difference 0, 02- Ars central to Taylor expansion-based extrapolation (Equation (4)), may not capture a meaningful direction not aligned. Consequently, naively constructed parameter-time trajectory f(t) can contain unknown noise or discontinuities. This makes it difficult to discern any true underlying temporal structure suitable for reliable interpolation extrapolation. We provide conceptual illustration of this issue with synthetic data in Appendix A.
of change if the parameters
To address this challenge, we adopt sequential fine-tuning, a specific instance of continual learning (CL) for training models over time. Specifically, when training the model for time step / on dataset Dr, we initialize its parameters with 0 the optimized parameters obtained from training on D;-1- The model is then fine-tuned:
0. arg min 6:
Σ CB(s(x);0)..v) with 0g initialized from Oz-1- (ESDE
(5)
By initializing from the previous solution, we show that consecutive parameters stay close to each other (Figure 7), aiding parameter interpolation and extrapolation. We remark that our goal of interpolation and extrapolation is different from prior interpolation studies (Hecht-Nielsen, 1990; Sussmann, 1992; Phuong and Lampert, 2020; Chen al., 2024) that interpolate path between two known model optima trained on the same dataset. We find path to an unknown future model state, which is exceptionally difficult. This is because constructing a non- non-linear path would require strong, and typically unavailable, assumptions about the evolution of data distribution with time.
4.2 CHALLENGES IN HYPERPARAMETER TUNING
The efficacy of parameter interpolation and extrapolation methods heavily relies on selecting appro- priate hyperparameters, such as the coefficients a in Equation (1), Equation (2), and Equation (4). In temporal generalization, this process presents a distinct challenge as hyperparameters must be chosen using only historical data. Any use of future data, which is reserved for evaluation is method- ologically unsound. Such practice would lead to overly optimistic performance assessments that do not reflect true generalization capabilities (Nylund al., 2024: Cha and Cho, 2024).
We sequentially tune the hyperparameter a by emulating a deployment scenario. At each time step t, with current model parameters 0, and data D., we determine the a to generate 0.48. Particularly, we simulate the hyperparameter choice we would have made at the previous step and find the optimal a" by assessing performance using metrie L on validation subset of the current data, Dr This involves generating candidate parameters (a) using past parameters and candidate value a*:
a' = arg min L (s (=:o(a))) .Daa) aES
Here, (a( are the parameters generated for this validation. The set S represents the defined search space for a, specific to the temporal generalization method (e.g., the interval [0, 1] for downscaling. and IR for extrapolation). This process is repeated new data becomes available and the resulting a* is then used with the current parameters 0, generate O.ts for upcoming periods.

---

Published as a conference paper at ICLR 2026
News summarization
Language modeling
36
Perplexity
34
Perplexity
6.5
6.0
5.5
30
1 2
7 8 9
10 11
12
5
6
7
8
9
10 11 12
13
Time (future)
Time (future)
Oracle (30.02=0.02)
Average 04.002=0.02)
Oracle (5.5 15.53+0.00
Average (6.3720.04)
Taylor (35.69=0.05)
Downscaling (32.3720.02)
Taylor (6.42200.12)
Downscaling (5.82 20.05)
Recent (32.51:0.02)
Recent (6.05=0.07)
Figure 2: Average perplexity comparison for T5-small model. Results for language modeling are
in the left figure, and those for news summarization evaluated over 12 future months (x-axis), with lower perplexity (y-axis) indicating better perfor- mance. Downscaling is the only method that did not lead to decrease in performance. Oracle contains complete knowledge of the future, trained and evaluated on data from time t.
in the right. Every month, cach method
GroupDRO
AVERACE
80
TRM
DOWNSCALING
Accuracy (%)
DeepCoral
RECENT
AGEM
60
40
20
Yearbook
HuffPost.
arXiv
FMoW
Figure 3: Comparison of accuracy for Wilds-Time datasets. We compare ERM (trained union of data up to time t) with DG (GroupDRO (Sagawa et al., 2020), IRM (Arjovsky et al., 2019), Deep- Coral (Sun and Saenko, 2016)), CL methods (EWC (Kirkpatrick et al., 2017), AGEM (Chaudhry et al., 2019)), parameter interpolation (Average, Downscaling, Recent), and Taylor extrapolation over 8 future months. No method consistently outperforms others across datasets. The details for the methods are provided in Appendix E.
EXPERIMENTAL RESULTS
We experiment with NewsRoom dataset (Grusky et al., 2018) to solve the language modeling and news summarization tasks using the T5-models (Raffel et al., 2020) in Figure 2. Addi- tionally, we use Wilds-Time (Yao al., 2022a) benchmark containing Yearbook (Ginosar et al., 2015), FMoW (Christie et al., 2018; Koh et al., 2021), HuffPost (Misra and Grover, 2021) and arXiv (Clement al., 2019) datasets Figure 3. As detailed Appendix C. a primary constraint for temporal generalization is the lack of public datasets with fine-grained temporal resolution. Most benchmarks offer only few coarse time points (e.g., yearly data), which insufficient mean- ingful extrapolation. The NewsRoom dataset is benchmark that contains monthly granularity.
We measure temporal generalization with 8-forward transfer (Lopez-Paz and Ranzato, 2017; Yao et al., 2022a), where we evaluate on å future time-stamps for all the datasets. Details regarding datasets, evaluation metrics, hyperparameter settings and models are in Appendix C, Appendix D, and Appendix E, respectively. Additional results are in Appendix F. Takeaway from our experiments that the inconsistent performance benchmarked approaches across multiple datasets. Based on evaluation, present four primary findings:

---

Published as a conference paper at ICLR 2026
News summarization
0.4
1003.0
Continual learning
0.2
Pre-trained initialization
Paremeter norm
0.0
1002.8
0.2
-0.4
1002.6
0066
1002.4
0.8
2012-01
2012-07
2013-01
2013-07
2014-01
2014-07
2015-01
2015-07
2016-01
2016-07
-1.0
2012
2013
2014
2015
2016
Time
Time
Figure 4: Effect of downscaling, The L2 norm of model parameters increases over time der continual learning, while it remains flat for models reinitialized each step. Downscaling reduces this overconfidence and improves tem- poral generalization.
Figure 5: Effect of extrapolation factor. The optimal extrapolation factor fluctuates with time and is often less than one or even nega- tive. This indicates that sometimes interpola- tion is preferable over extrapolation for tempo- ral generalization.
50
Yearbook
Yearbook
Perplexity
5.0
20
2000
2.5
40
PC2
UMAP 2
1980
35
-2.5
0
1960
1
2
3
4
5
6
8
9
10
11
12
13
-5.0
1940
Time (future)
-10
10
Average (CL) (34.0)
Average (Pre) (39.5)
PC
20
UMAP
Downscaling (CL) (32.5) -
Downscaling (Pre) (40.6)
Figure 7: Dimensionality reduction. PCA (left) and UMAP (right) on models trained with year- book dataset showing the change of parameters with time.
Taylor (CL) (35.7)
Taylor (Pre) (52.0)
Figure Effect of CL. For language mod- elling, CL. (solid lines) improves perplexity compared to pre-trained model (dashed lines).
1) No method consistently improves performance over the recent model. Across all datasets, our results show that most methods degraded performance compared to using only the most recent model. The performance decline with model averaging s likely due to older parameters introducing noise as a result of distributional shifts over time. Another key observation is that downscaling the most recent parameters with single scalar reduces perplexity for language modeling and news summarization, but gives sub-par performance on Wilds-Time datasets. As shown Figure 4, the norm of the parameters consistently grows time, potentially hurting generalization as discussed in Section 3. This observation supports the strategy of shrinking the parameter norm at inference (a 1), obtaining a similar (sometimes better) model performance for temporal generalization reduces the present model's I's overconfidence against an unpredictable future.
2) Taylor-Series extrapolation underperforms compared to other methods. Its performance was consistently lower than that of a recent model. This outcome contrasts with the successful extrapo- lation reported for Time-vectors by Nylund al. (2024) due the lack of access the future data in our setting. To show that this simple approach does not work, we tuned the parameter in Equa- tion (4) using future validation data each timestamp for the news summarization task (see Figure for trends). The optimal values for were frequently less than one negative. An optimal suggests that a dampened extrapolation is preferable, while a negative a indicates that adjusting in the direction opposite to the first-order estimated change yields better performance. This implies that interpolation is sometimes preferable over forward extrapolation for temporal generalization.
3) Continual learning is important. Figure 6 compares CL with training each timestamp model independently, initialized from a pre-trained T5- rm-ll checkpoint. The results demonstrate that CL improves forward transfer for both interpolation and extrapolation methods significantly. This rein-
8

---

Published as a conference paper at ICLR 2026
forces hypothesis that initializing each time step parameters with the previous one keeps param- eters close to each other and preserves a consistent trajectory in parameter space. This smoothing effect is further illustrated by comparing the principal component projections of parameter trajecto- ries: CL yields smooth, coherent paths (Figure F. 14)4, contrasting sharply with the disjoint and noisy trajectories observed when models are trained independently from a pre-trained model (Figure F.15).
We highlight that Figure 6 shows that sequential fine-tuning is necessary for any extrapolation method work. All methods perform better with CL backbone than with a pre-trained initial- ization because the parameters for adjacent time steps must lie close to each other. Figure 4 shows that this not a sufficient condition. While CL helps, can also lead an increase in the parameter norm, which is reduced by using downscaling as simple method.
6 LIMITATIONS AND FUTURE DIRECTIONS
F Fundamental constraints on temporal generalization. The challenge of temporal generalization subject to fundamental theoretical constraints, notably those underscored by the No Free Lunch theorem. Without strong and empirically validated assumptions regarding the nature of temporal distribution shifts, no algorithm can universally guarantee optimal future performance against arbi- trary changes. This principle highlights an inherent vulnerability of any method: an approach that performs well on certain future distributions may fail on others due to the inherently arbitrary and unpredictable nature of future distributional shifts.
Future directions. Given these theoretical constraints, key future direction to model the non-linear evolution of parameters with time by making explicit assumptions about how the data- generating process evolves over time. Our dimensional reduction analysis provides an empirical starting point in Figure 7, Figure F.13. We find one-dimensional structure corresponding to time, but the interaction of this dimension with the full parameter space remains an open prob- lem. As promising first step, begin by capturing this relationship with two approaches below: Learning the change (a) considers learning single offset 0 to capture parameter changes:
Table 1: Perplexity compari- son of Taylor expansion with (a) and (b).
ΣΣ 00. - min 92
+ +0-=:+00)11, + A 1101122
(a)
(b) Future (12 months)
8=0
6.05g.07 13.06g.17
where 0 represents the learned change in parameters. We include a regularization term |00|| to find the minimum parameter change.
6.09g.09/ 13. 11.007..00 07..1 6.07a.09/ 13. 07..17
Learning the coefficient (b) allows the extrapolation multiple time periods into the future by learning the coefficient as:
ΣΣ min 8 0,,8
0, (0--0: + softplus(a5 33) 20000)1 + > 11110112.
(8)
8=
where softplus(a + 3) provides smooth, positive scaling factor that scales the magnitude of AA. Despite enforcing minimal parameter change and allowing for a time-varying scale, Table 1 shows that we underperform the most recent model. The challenge lies in understanding the interplay between millions of parameters. While parameter extrapolation seems appealing, real-world data needs conservative approach for temporal generalization.
CONCLUSION
Our work highlights the challenges and pitfalls in temporal generalization, an area where prior work has made strong claims of generalization. We systematically investigated this problem through the perspectives of interpolation and extrapolation of parameters without access to future data. Our analysis across multiple datasets demonstrates the absence of a superior method compared to the most recent model, underscoring the nuanced and context-depender nature of this problem. Our findings highlight that the key temporal generalization is thus not to design algorithms, but to identify the reasonable assumptions about how the data generating process evolves over time. Only by making those assumptions explicit can we hope develop methods that generalize over time.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) with a grant funded by the Ministry of Science and ICT (MSIT) of the Republic of Korea in connection with the Global AI Frontier Lab International Collaborative Research, Sam- sung Advanced Institute of Technology (under the project Next Generation Deep Learning: From Pattern Recognition to AI), National Science Foundation (NSF) award No. 1922658, Center for Advanced Imaging Innovation and Research (CAI2R), National Center for Biomedical Imaging and Bioengineering operated by NYU Langone Health, and National Institute of Biomedical Imaging and Bioengineering through award number P41EB01718% 7183. The computational requirements for this work were supported by NYU IT High Performance Computing resources, services, and staff expertise and NYU Langone High Performance Computing Core's s resources and personnel. This work was partly supported in part by the NYUAD Center for Interdisciplinary Data Science & AI (CIDSAI), funded by Tamkeen under the NYUAD Research Institute Award CG016. This content solely the responsibility of the authors and does not represent the views of the funding agencies.
REFERENCES
J. Achiam, S. Adler, Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 190.00990,,,289,, 2019. 3, 7, 19
G. Bai, C. Ling, and L. Zhao. Temporal domain generalization with drift-aware dynamic neural networks. In Proceedings of the International Conference Learning Representations (ICLR), 2023. 2, 3, 16
L. Barrault, M. Biesialska, O. Bojar, M. R. Costa-jussà, C. Federmann, Y. Graham. R. Grundkiewicz, B. Haddow, M. Huck, E. Joanis, T. Koemi, P. Koehn, C.-k. Lo, N. Ljubešić, C. Monz, M. Mor- ishita, M. Nagata, T. Nakazawa, S. Pal, M. Post, and M. Zampieri. Findings of the conference on machine translation. In Proceedings of the Conference on Machine Translation (WMT), 2020. 17
Z. Cai, G. Bai, R. Jiang, X. Song, and L. Zhao. Continuous temporal domain generalization. arXiv preprint ar X1v:24405.16075 2024. 2, 3, 16
S. Cha and K. Cho. Hyperparameters in continual learning: a reality check. Transactions on Machine Learning Research (TMLR), 2024. 2, 6
A. Chaudhry, M. Ranzato, M. Rohrbach, and M. Elhoseiny. Efficient lifelong learning with a-GEM. In Proceedings of the International Conference on Learning Representations (ICLR), 2019. 7, 19
A. Chen, R. Shwartz-Ziv, Ziv, K. Cho, M. L. Leavitt, and N. Saphra. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in MLMs. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. 6
G. Christie, N. Fendley, J. Wilson, and R. Mukherjee. Functional map of the world. In Proceedings the the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 7. 17, 19
C. B. Clement, M. Bierbaum, K. P. O'Keeffe, and A. A. Alemi. On the use of arxiv as a dataset. arXiv preprint
pee:::905000077 2019. 7, 17, 19
A. Cohan and N. Goharian. Revisiting summarization evaluation for scientific articles. In Pro- ceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2016. 20
B. Cottier, R. Rahman, L. Fattorini, N. Maslej, T. Besiroglu, and D. Owen. The rising costs of training frontier ai models. arXiv preprint arXiv:2405.1015, 2024. 1
H. Dai, R. Techan, and M. Ren. Are llms prescient? a continuous evaluation using daily news as the oracle. arXiv preprint 1..02424 2024. 3,
10

---

Published as a conference paper at ICLR 2026
M. Davari and E. Belilovsky. Model breadcrumbs: Scaling multi-task model merging with sparse masks, In Proceedings of the European Conference on Computer Vision (ECCV), 2024. 2,
S. Dohare, J. F. Hernandez-Garcia, Q. Lan, P. Rahman, A. R. Mahmood, and R. S. Sutton. Loss of plasticity in deep continual learning. Nature, 2024. 5
J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 2011. 17
Dziadzio, V. Udandarao, Roth, A. Prabhu, Z. Akata, S. Albanie, and M. Bethge. How to merge your multimodal models over time? arXiv preprint arXiv:2412.06772, 2024. 2,
J. Fang, H. Jiang, K. Wang, Y. Ma, J. Shi, X. Wang, X. He, and T.-S. Chua. Alphaedit: Null-space constrained model editing for language models. In Proceedings of the International Conference Learning Representations (ICLR), 2025. 4
P. Foret, A. Kleiner, H. Mobahi. and B. Neyshabur. Sharpness-awaad minimization for efficiently improving generalization. In Proceedings of the International Conference on Learning Represen- tations (ICLR), 2024. 5
S. Ginosar, K. Rakelly, S. Sachs, B. Yin, and A. A. Efros. A century of portraits: A visual historical record of american high school yearbooks. In Proceedings of the IEEE International Conference Computer Vision Workshops, 2015. 7, 17, 19
T. Goyal, J. J. Li, and G. Durrett. News summarization and evaluation in the era of gpt-3. arXiv preprint arXiv:2209.123:: 2022. 20
M. Grusky, Rogue scores. A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2023. 20
M. Grusky, M. Naaman, and Y. Artzi. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL), 2018. 7, 17, 18
E. Hazan et al. Introduction to online convex optimization. Foundations and Trends@ in Optimiza- tion, 2016. 17
R. Hecht-Nielsen. On the algebraic structure of feedforward network weight spaces. Advanced Neural Computers, 1990. 2,
E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. S. Wang, L. Wang, W. Chen, et al. Lora: Low-rank adaptation of large language models. In Proceedings of the International Conference on Learning Representations (ICLR), 2022. 19
G. Huang. Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 3, 19
G. Ilharco, M. Wortsman, S. Y. Gadre, S. Song. H. Hajishirzi, S. Korbblith, A. Farhadi, and L. Schmidt. Patching opca-vocy models by interpolating weights. In Advances in Neu- ral Information Processing Systems (NeurIPS), 2022. 2,
G. Ilharco, M. T. Ribeiro, M. Wortsman, S. Gururangan, L. Schmidt, H. Hajishirzi, and A. Farhadi. Editing models with task arithmetic. In Proceedings of the International Conference on Learning Representations (ICLR), 2023. 4
K. Jaidka, N. Chhaya, and L. Ungar. Diachronic degradation of language models: Insights from social media. In Proceedings of the Annual Meeting of the Association for Computational Lin- guistics (ACL), 2018. 1
D.-H. Jang, S. Yun, and D. Han. Model stock: All need is just few fine-tuned models. In Proceedings of the European Conference on Computer Vision (ECCV), 2024. 2, 4
Z. Ji and M. Telgarsky. Directional convergence and alignment in deep learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.

---

Published as a conference paper at ICLR 2026
D. P. Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
19
J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinkka, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017. 7, 19
P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang. A. Balsubramani, W. Hu, M. Yasunaga, R. L.. Phillips, I. Gao, et al. Wilds: benchmark of in in-the-wild distribution shifts, In Proceedings of the International Conference on Machine Learning (ICML), 2021. 7
A. Lazaridou, A. Kuncoro, E. Gribovskaya, D. Agrawal, A. Liska, T. Terzi, M. Gimenez, C. de Mas- son d' Autume, T. Kocisky, S. Ruder, et al. Mind the gap: Assessing temporal generalization in neural language models. In Advances in Neural Information Processing Systems (NeurlPS), 2021.
B. Lester, R. AI-Rfou, and N. Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.0869) 2021. 19
A. Lewandowski, M. Bortkiewicz, S. Kumar, A. György, D. Schuurmans, M. Ostaszewski, and M. C. Machado. Learning continually by spectral regularization. In Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2025.
and Arora. An exponential learning rate schedule for deep learning. In Proceedings of the International Conference Learning Representations (ICLR), 2019. 5
D. Lopez-Paz and M. Ranzato. Gradient episodic memory for continual learning. Advances in Neural Information Processing Systems (NeurIPS), 2017. 7, 18
I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2019, 18, 19
K. Luu, Khashabi, S. Gururangan, K. Mandyam, and N. Smith. Time waits for no one! analysis and challenges of temporal misalignment. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2021. 1,
W. Merrill, V. Ramanujan, Y. Goldberg, R. Schwartz, and N. A. Smith. Effects of parameter norm growth during transformer training: Inductive bias from gradient descent. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
R. Misra and J. Grover. Sculpting Data for ML: The first act of Machine Learning. 2021. ISBN
778-0-578-83125-1. 7. 17, 19
A. Nasery, S. Thakur, V. Piratla, A. De, and Sarawagi. Training for the future: simple gradient interpolation loss to generalize along time. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 2,5, 17
E. Nikishin, M. Schwarzer, P. D'Oro, P.-L. Bacon, and A. Courville. The primacy bias in deep reinforcement learning. In Proceedings of the International Conference Machine Learning (ICML). 2022. 5
K. Nylund, S. Gururangan, and N. Smith. Time encoded in the weights of finetuned language models. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2024. 1, 2, 3, 4, 6, 8, 17, 18
M. Phuong and C. H. Lampert. Functional vs. parametric equivalence of re( networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2020. 2, 6
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 2020. 3, 7, 18
A. Rakhlin and K. Sridharan. Online learning with predictable sequences. Proceedings of the Annual Conference Learning Theory (COLT), 2013. 17
12

---

Published as a conference paper at ICLR 2026
K. Roth, V. Udandarao, S. Dziadzio, A. Prabhu, M. Cherti, O. Vinyals, O. J. Henaff, S. Albanie, M. Bethge, and Z. Akata. A practitioner's ' guide to continual multimodal pretraining. In Advances in Neural Information Processing Systems (NeurlPS), 2024. 3
S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In Proceedings of the International Conference on Learning Representations (ICLR), 2020. 3, 4, 7, 19
V. Sanh, L. Debut, J. Chaumond, and T. Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 108, 2019. 19
T. Sellam, D. Das, and A. Parikh, Bleurt: Learning robust metrics for text generation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL.), 2020. 20
B. Sun and K. Saenko. Deep coral: Correlation alignment for deep domain adaptation. Proceed- ings of the European Conference Computer Vision (ECCV), 2016. 3, 7, 19
H. J. Sussmann. Uniqueness of the weights for minimal feedforward nets with a given input-output map. Neural Networks, 1992. 2, 6
G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint
MxXiv:2312.11005 2023. 1
P. Wang. Z. Li, N. Zhang, Z. Xu, Y. Yao, Y. Jiang, P. Xie, Huang, and H. Chen. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
P. Yadav, D. Tam, L. Choshen, C. A. Raffel, and M. Bansal. Ties-merging: Resolving interference when merging models. In Advances in Neural Information Processing Systems (NeurIPS), 2024. 2,
M. Yamada, T. Yamashita, S. Yamaguchi, and D. Chijiwa. Toward data efficient model merging between different datasets without performance degradation. In Proceedings of the Asian Confer- ence on Machine Learning (ACMI), 2025. 4
H. Yao, C. Choi, B. Cao, Y. Lee, P. W. W. Koh, and C. Finn. Wild-time: A benchmark of in-the-wild distribution shift over time. Advances Neural Information Prucessing Systems (NeurIPS), 2022a. 4, 7, 17, 18, 19
H. Yao, Y. Wang. S. Li, L. Zhang, W. Liang, J. Zou, and C. Finn. Improving out-of-distribtiion ro- bustness via selective augmentation. In Proceedings of the International Conference on Machine Learning (ICML), 2022b.
M. Yashwanth, G. K. Nayak, H. Rangwani, A. Singh, R. V. Babu, and A. Chakraborty. Minimizing layerwise activation norm improves generalization in federated learning. In Proceedings of the Winter Conference on Applications of Computer Vision (WACV), 2024. 5
X. Yue, Y. Zhang, S. Zhao, A. Sangiovanni- Vincentelli.nnnlli.. K. Keutzer, and B. Gong. Domain ran- domization and pyramid consistency: Simulation- -t-relll generalization without accessing target domain data. In Proceedings of the International Conference on Computer Vision (ICCV), 2019.
3
J. Zhang, A. K. Menon, A. Veit, S. Bhojanapalli, S. Kumar, and S. Sra. Coping with label shift via distributionally robust optimisation. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi Bertscore: Exaluating text generation with bert. In Proceedings of the International Conference on Learning Representations (ICLR), 2020. 20
Y. Zhao, H. Zhang, and X. Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. In Proceedings of the International Conference on Machine Learning (ICML), 2022. 5
13

---

Published as a conference paper at ICLR 2026
C. Zhou, X. Ma, P. Michel, and G. Neubig. Examining and combating spurious features under distribution shift. Proceedings of the International Conference on Machine Learning (ICML), 2021. 3
K. Zhou, Y. Yang. T. Hospedales, and T. Xiang. Deep domain-adversarial image generation for domain generalisation. In Proceedings of the AAAI National Conference on Artificial Intelligence (AAAI), 2020.
C. Zhu, N. Chen, Y. Gao, and B. Wang. Is your Ilm outdated? a deep look temporal generalization. In Proceedings of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL), 2025.
M. Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Pro- ceedings of the International Conference Machine Learning (ICML), 2003. 17
14

---

Published as a conference paper at ICLR 2026
APPENDIX
Organization. In the appendix, we discuss the challenge of non-convexity for temporal gener- alization (Appendix A), the datasets used and their specifics (Appendix C), the evaluation metrics applied (Appendix D), the hyperparameter configurations for our experiments (Appendix E), and additional experimental results (Appendix F).
A CHALLENGE OF NON-CONVEXITY FOR TEMPORAL GENERALIZATION
We define model's parameters 0, as identifiable if distinct parameter values always yield distinct predictive functions:
- -
= 0).4) B!88
(9)
for any two parameterizations 0/-4) and 0( where f(:;0.) is the model's predictive function. different learned parameters O, result in functionally identical models, the parameter-time function
be noisy, posing significant challenge in extrapolating these parameters to 0.+8- In this section, we highlight these issues within controlled synthetic time-varying regression setup, contrasting linear and non-linear models.
Synthetic Experimental Setup. To isolate identifiability effects, we consider input features X, R&d sampled independently at each time step from x, NN(,,) The underlying true data-generating parameters Or E Rd evolve via cubic polynomial:
0r = a + bt + cl + dd"
(10)
where a, b, c, d E IRd are fixed coefficients. The observed continuous target is generated by:
V/ 0g + Et:
with E NO, o²)) being i.i.d. Gaussian noise. We compare two modeling approaches:
1. Linear Regression Model predicts ye using linear function of the inputs,
Tr = x O..
(12)
2. Non-linear Regression Model uses non-linear model, such Multi-Layer Perceptron (MLP) with parameters 0,:
Te (x):;))
(13)
For both models, we learn parameters 0g from sequences (X1, We) and evaluate on the future data.
Identifiability and Extrapolation in Linear Models. Linear regression yields parameters 6 that consistently converge to the true parameters 07- This implies that both the magnitude and the direc- tion of 07 are identifiable via 0.. Figure A.8 shows that the Taylor series expansion of the learned tra- jectory O, produces reliable estimates of future parameters 0.+6- Consequently, the model achieves better temporal generalization, maintaining predictive accuracy on future data, as shown Fig- ure A0
Identifiability Challenges and Extrapolation in Non-linear Models. Non-linear models incur identifiability challenges with their parameters 0,. Even if we accurately learn the input-output re- lationship, the MLP parameters O, may not be unique. This lack of uniqueness stems from several factors: (i) Symmetries: Many non-linear architectures possess inherent symmetries (for example, permutation of hidden units in an MLP) where different 0, yield identical functions. (ii) Non-conver Optimization: The optimization landscapes for these models typically nonconvex, meaning the optimizers may converge to different local minima, each corresponding to distinct 9 that are func- tionally similar the training distribution but structurally different.
These identifiability issues imply that the specific learned parameter trajectory 6, might one of many possible trajectories, exhibiting noisy and unstable behavior over time. Figure A.9 shows the
15

---

Published as a conference paper at ICLR 2026
0
01
True 0
6
True 6
Parameter Value
Parameter Value
Learned
Learned 6:
1
Taylor 02
Taylor O2
4
-2
2
-3
0
2
4
Time (t)
Time (t)
Figure A.8: Comparison of true underlying parameters 07, Iearned parameters 0, of linear model and Taylor second-order extrapolated parameters d.. The identical plots for the true parameters and extrapolated parameters illustrate effective parameter estimation.
25
Oracks Taylor 02
Log MSE
102
20
10
[011
10-2
1.0
5
Evaluation Time (t/+1)
Linear Oracle
MLP Oracle
0
Linear Boseline
MLP Baseline
Time
Taylor
Taylor
Linear
Taylor
Figure A.9: MLP Parameter Norm Evolution: l norm comparison of MLP Oracle, and extrap- olated parameters using Taylor O1 (first-odder) and 02 (second-order) approximations. The di- vergence in extrapolated norms highlights chal- lenges in extrapolating ML.P parameter trajecto- ries across time.
Figure A.10: Log MSE Extrapolation Perfor- mance: One-step-ahead performance compar- ison for linear (solid lines) and MLP (dashed lines) models. The linear model with identifi- able parameters allows for more effective Taylor extrapolation than MLP, whose parameter non- identifiability impedes performance. The oracle performanace linear and MLP overlaps.
l2 divergence of Taylor approximation from the true model. Consequently, directly extrapolating such 0r trajectory to obtain future parameters 6.-s becomes highly unreliable compared to linear model as shown Figure A. This inability fundamentally limits the ability temporal gener- alization and contrasts sharply with well-specified linear models where parameter estimate 0,48 is meaningful.
B RELATED WORK
Temporal Generalization. Primary challenges for temporal generalization methods are compu- tational scalability and data scarcity, which render them impractical for large-scale models. Ap- proaches that train auxiliary models over the parameters of network, such as recurrent networks (Bai et al., 2023) or autoencoders (Cai et al., 2024), exemplify this problem. The recurrent method of Bai et al. (2023) has a complexity of O O(Nd+C) with N being the parameter count of the predictive model, d the width of the last hidden layer, and C' the parameter count of preceding layers. Its linear scaling with is intractable for large models, especially those with nigh-dimensional outputs such language vocabulary. Similarly, training autoencoder the full parameter set (Cai al.,
16

---

Published as a conference paper at ICLR 2026
2024) of widely used architectures is computationally infeasible. The coarse granularity of available temporal data, often limited to yearly checkpoints, makes learning the underlying dynamics chal- lenging and undermines the justification for such data-intensive approaches. The approach in Nasery et al. (2021) models the function's 's output with respect to a time input, which necessitates intrusive architectural changes and full model fine-tuning. For practical applications, we avoid the need for expensive auxiliary models, architectural modifications, and reliance on fine-grained temporal data.
Online learning. Our problem setting also relates to, yet differs significantly from, online learning. Online learning emphasizes rapid adaptation to streaming data, where a model sequentially pre- dicts, observes an outcome, and updates to minimize cumulative regret over time, focusing on how quickly it can converge to the best dynamic solution (Zinkevich, 2003; Duchi et al., 2011; Rakhlin and Sricharan, 2013; Hazan et al., 2016). In contrast, temporal generalization assesses model's performance on unseen future data using parameters at time f, without any adaptation during this future deployment. Thus, while online learning is concerned with the efficiency of continuous adap- tation within an immediate predict-observe-update loop, our focus is on the a priori generalization to future distributions. The challenge in temporal generalization lies in necessitating assumptions about how the future might relate the past, a consideration less central to the regret minimization framework of online learning.
C DATASETS
This section details the datasets used in our work. These datasets were selected primarily for their distinct temporal characteristics. Collectively, they span a diverse range of tasks and data modalities, which facilitates a thorough investigation of model behavior over time:
NewsRoom (Grusky et al., 2018): This dataset is a large corpus comprising approximately 9 million news articles published between 2009 and 2016. Following Luu et al. (2021); Nylund al. (2024), we partition the data by month and year to evaluate both language modeling and news summarization tasks. For language modeling, we specifically focus on the 2012-2016 period, utilizing the English subset of the WMT news dataset (Barrault et al., 2020). From this WMT subset, we sample approximately 7.1 million tokens from articles each month for training and 700k-7200 tokens for testing per month. WMT training and testing splits for August 2012 and May 2016 were unavailable. For the news summarization task (NewsSum), our setup is based on the original task defined by Grusky et al. (); Nylund et al. (2024) with the post-processing steps detailed by Luu et al. (2021). To the best of our knowledge, NewsRoom is distinct in offering data at this scale for monthly temporal evaluations, providing approximately 60 months (5 years) of continuous data from 2012-2016 suitable for this kind of analysis for temporal generalization.
WILDS- (Yao et al., 2022a): This benchmark comprises multiple, temporally-struucurre datasets spanning diverse modalities. In this work, we focus on four specific datasets from this benchmark: the image-based datasets Yearbook and FMoW, and the text-based datasets ArXiv and HuffPost.
Yearbook (Ginosar et al., 2015): This dataset consists of 37k grayscale portrait photographs from U.S. high school yearbooks between 1930 and 2013. The task is to predict binary gender labels from facial features. Faces and fashion styles evolve significantly over time, making this useful benchmark for evaluating visual domain shifts.
HuffPost (Misra and Grover, 2021): This dataset consists news headlines published between 2012 and 2018, each labeled with one of several topic categories (c.g., politics, entertainment, technology). The task is to classify headlines, thereby simulating topic identification chal- lenges under the influence of evolving media discourse and shifting language use over time.
arXiv (Clement et al., 2019): Comprising paper titles submitted to arXiv from 2007 to 2022 (2M examples), the goal is to predict the subject category (e.g., CS.LG, math.AP). The tem- poral challenge stems from the gradual evolution of scientific language, occasionally marked by more abrupt shifts due the emergence of new research fields or terminology.
FMoW (Functional Map of the World) (Christie et al., 2018): The FMoW dataset a large- scale remote sensing collection of approximately 119,000 satellite images gathered between 2002 and 2017 annotated with land-use classes (c.g., airport, forest, hospital). Changes in infrastructure and land use create potential temporal shifts in the data distribution.
17

---

Published as a conference paper at ICLR 2026
Table E.2: Summary of Model Architectures and Hyperparameters
Dataset
Task
Model
Optimizer
LR
Batch Size
Epuchs/tteratioss
Nowm
Language Modeling
T5
Adamm
8 10
16
epoch
months
NewsRoom
News Summarization
T5
AdamW
8 10
16
epochs
12 months
Yearbook
Gender Prediction
4-1ayer CNN
Adam
1xx
32
300 iterations
10 years
HuffPost
Hcadline Topic Classification
Dist
AdamW
2 10-5
32
1000 iterations
3 years
ArXiv
Paper Subject Prediction
DistiBERT
Adam W
2xx 10-5
64
1000
iterations
6 years
FMoW
Image Classification
Densenet- 121
Adamm
1
64
500
iterations
years
D METRICS FOR TEMPORAL GENERALIZATION
To evaluate how well a model, trained at specific time 6, generalizes to future time intervals, we use the 6-forward transfer (FWT) framework (Lopez-Paz and Ranzato, 2017; Yao et al., 2022a). Let Twai be the set of training timestamps for which forward transfer is evaluated. Given the sequence of total datasets (Delr=1 (as defined previously), this set of training timestamps is Tirin {t 1 I 1 1 T 6}. Let Me. denote the performance (e.g., accuracy) of the model associated with training timestamp E T Tirinn (e.g- 0z), when evaluated on data from a future timestamp j. The average and worst-case forward transfer metrics are defined as follows.
Average FWT (Avgewt) aggregates the performance on all valid future datasets within d-horizon:
AvgewT
Mr
(14)
Neval tETim
where t + l denotes the k-th future timestamp relative to t. Noval is the total number of M terms included in the sum (i.e., the count of valid (f, t-+ k) pairs for which performance is measured within the specified horizon). If for every & E Train. all future evaluation points (t ( l,..., are available and valid, then Neval [Tuaial 8.
Worst-case FWT (Worstewr) captures the worst-case performance of each model among its fu- ture evaluations within the d-horizon. This is then averaged over training timestamps in Train- Assuming M represents accuracy (where higher is better), we can write it formally as follows:
Worstewt
Σ
min (E(......8)
Me,++K
(15)
[Train
BETTun
If, for given t, Mr, is not available for all & in this range (e.g., if + T)), the min should be taken over the subset of these steps for which Me,++ is validly defined. If the performance metric M were an error rate (where lower better), the min operator would be replaced by max.
The min operation is over the set of & future steps {1,...,á} d}.
Unlike traditional continual learning, which focuses mitigating catastrophic forgetting (i.c., main- taining performance on previously learned tasks), our focus here is on forward transfer: assessing how well models trained past data generalize future, unseen time horizons.
E HYPER-PARAMETERS AND MODEL ARCHITECTURES
This section details the model architectures and hyperparameters used for our experiments. For the NewsRoom dataset, our setup largely follows Nylund et al. (2024), while for the WILDS-Time datasets (Yearbook, HuffPost, ArXiv, and FMoW), we adhere the experimental configurations outlined in Yao et al. (2022a), unless specified otherwise. A summary is provided in Table E.2, with detailed descriptions following.
E. GENERAL HYPER- HYPER-PARAMETER AND MODEL ARCHITECTURES
NewsRoom dataset For the NewsRoom dataset (Grusky al., 2018), used T5-small with 70 million parameters and T5-lerge with 770 million parameters (Raffel et al., 2020). Both models were trained with the AdamW optimizer (Loshchilov and Hutter, 2019) using a learning rate of
18

---

Published as a conference paper at ICLR 2026
8 x 10-4, 4, batch size of 2, and gradient accumulation steps, resulting in an effective batch size of 16. For language modeling, we used the LM adaptation objective (Lester et al., 2021). The T5-large model was fine-tuned using Low- Rank Adaptation (LoRA (Hu et al., 2022)) with parameters r = 8, a = 32, and dr.pout=0.1. LoRA was applied to the query (q) and value (v) attention modules. We used one epoch of training for language modeling and three epochs for downstream tasks.
Yearbook dataset For experiments on the Yearbook dataset (Ginosar et al., 2015), we used a 4-layer Convolutional Neural Network (CNN). The model was trained using the Adam optimizer (Kingma, 2014) with a learning rate of 1 x 10-3 and a batch size of 32. For each timestamp, the network was trained for 300 iterations and evaluated for gender prediction for ten subsequent years.
HuffPost dataset Experiments on the HuffPost dataset (Misra and Grover, 2021) were conducted using an uncased DistilBERT model (Sanh al., 2019), augmented with fully-connected classi- fication layer. The AdamW optimizer (Loshchilov and Hutter, 2019) was used for training, with a learning rate of x 10-5 and a batch size of 32. Models were trained for 1000 iterations for each timestamp. Performance was evaluated based on predictions for the three years following each timestamp.
Xrx dataset The experimental setup for the ArXiv dataset (Clement et al., 2019) mirrored that of the HuffPost dataset in terms of model architecture, utilizing an uncased DistiBBERT model (Sanh et al., 2019) with appended fully-connected classification layer. Training the A optimizer (Loshchilov and Hutter, 2019) with a learning rate of 2 x 10-5, For this dataset, batch size of 64 was used, and the model was trained for 1000 iterations. Evaluation was conducted on data from the next six years.
performed using
FMoW dataset For the FMoW dataset (Christie et al., 2018), we used a Densenet-122 121 architecture (Huang et al., 2017), which was pre-trained on the ImageNet dataset. The Adam optimizer was used to train the model with learning rate of 10-1 and a batch size of 64. The training process consisted of 500 iterations. Notably, no L2 regularization was applied during the training of this model. Evaluation was performed on data corresponding the six subsequent years.
E.2 BETHOD-SPECIRES
Method details.
We provide a description of the methods and their hyper-parameters below:
Empirical risk minimization (ERM) in our sequential setting involves training the model at each time / on the cumulative union of all data observed from the start up to time I.
Invariant risk minimization (IRM) (Arjovsky et al., 2019) attempts to discover invari- ant predictor by learning a data representation where the classifier is consistent across all training domains. The goal is to capture causal mechanisms that are stable across environ- ments while disregarding spurious correlations. We employ the penalty-based approxima- tion with a penalty of 1.0 for our experiments (Arjovsky al., 2019; Yao et al., 2022a).
Group distributionally robust optimization (GroupDRO) (Sagawa et al., 2020) opti- mizes for the worst-case loss encountered in the data from previous time stamps. We follow the standard implementation from previous works (Sagawa et al., 2020; Yao et al., 2022a).
Deep correlation alignment COeePCORAL) (Sun and Saenko, 2016) adds a loss term that aligns the second-order statistics of the feature distributions between data from a source time period and a target time period. We follow the implementation from prior works (Sun and Saenko, 2016; Yao et al., 2022a) and use CORAL penalty of 0.9 for all datasets.
Averaged gradient episodic memory (A-GEM) (Chaudhry et al., 2019) is replay- based continual learning method that uses an episodic memory store a small number of rep- resentative examples from past timestamps. Following prior works (Yao et al., 2022a), a buffer size of 1000 used for all datasets
Elastic weight consolidation (EWC) (Kirkpatrick et al., 2017) is regularization based continual learning method that uses a regularization term to penalize significant alterations to model parameters, anchoring the model to previously learned knowledge about past dynamics. A regularization strength of 0.5 was used across all datasets (Yao et al., 2022a).
For the downscaling method, we used a = 0.956892 for T5-Small (for all timestamps in both language modeling and news summarization tasks the NewsRoom dataset) and a 0.988181
19

---

Published as a conference paper at ICLR 2026
News summarization
News summarization
6.8
4.0
6.6
Perplexity
3.8
Perplexity
6.4
3.6
6.2
6.0
3.4
5.8
9 10 11 12
Time (future)
Time (future)
Oracle (3.45=0.02)
Average (3.78=0.02)
Oracle (5.87=0.05)
Average (6.45=0.04)
Taylor (3..8800.44
Downscaling 0..24
Taylor (6.63:00.9)
Downscaling 06.35=0.05)
Recent (3.67)0.03)
Recent (6. (6.42+0.06)
News summarization
News summarization
44
62
42
ROUGE-L
ROUGE-L
60
40
38
58
36
56
1
2
3
6
7
8
9
10
12
13
4
5
6
10 11 12 13
Time (future)
Time (future)
Oracle 9220.
Average (38.0220.60)
(61. 77.0.55)
Average
Taylor (39.59:1.21)
Downscaling (4095120.90)
Taylor (59.67+1.21)
Downscaling (60.13
Recent (40.5010.86)
Recent (60.2710.69)
Figure F.11: Perplexity and ROUGE-L comparison for T5-large model. We contrast average- case performance (left panels) with worst-case performance (right panels) over future time steps (x- axis). Effective summarization is marked by lower Perplexity scores and higher ROUGE-L scores (y-axis). Downscaling is the only method that does not consistently lead to performance decay both evaluation metrics compared to the recent model.
for Larger These values determined via random grid search over the interval [0.9, 1.0). To obtain the Taylor approximation in Equation (6), we conducted a random grid search with 30 values sampled from the interval [-1, 1]. For Wilds-Time, we found 1.0 be consistently optimal for the simulated future for the datasets. Each experiment was run five times to report the mean and standard deviation.
F ADDITIONAL EXPERIMENTAL RESULTS
Results with La-Lerge and T5-small model. The results for T5-large in Figure F.11 and T5-small in Figure F.12 largely affirm the key trends discussed in the main paper. None of the benchmarked methods including parameter averaging, first-order Taylor expansion and downscaling consistently outperform the recent model. Regarding the evaluation metrics, we contend that ROUGE-L presents considerable limitations for assessing temporal generalization. Its inherent sensitivity to lexical variations, rather than semantic shifts, can be misleading in this context. Many studies show an inconsistent correlation of ROUGE-L with human evaluations across different applications (Goyal al., 2022; Cohan and Goharian, 2016; Grusky, 2023). It may not adequately capture the nuanced improvements associated with language model scaling, where semantic fidelity often outweighs exact lexical replication (Sellam et al., 2020; Zhang et al., 2020).
20

---

Published as a conference paper at ICLR 2026
Language modeling
summarization
42
40
Perplexity
Perplexity
38
36
34
32
11
5 6
10 11 12 Time (future)
1
2
3
8 9 10 11 12 13 Time (future)
Dracle (33.32+0.03)
Average (38.25+0.00 62)
Dracle (11.15=0.08)
Average (13.16+0 051
Taylar (42.37:10.0))
Downscaling (38.1410.00)
Taylor (14.02.0.21)
Downscaling (12.4110.10)
Recent (38.32x0.03)
Recent (13.06x0.17)
(a) Worst-case Perplexity for T5-small model. The left panel shows results for language modeling and the right panel for news summarization. Each method is evaluated over future time steps (x-axis), with lower Perplexity values (y-axis) indicating better performance. Consistent with the average-case performance, that only downscaling does lead drop in performance compared recent model.
News summarization
News summarization
62
42
40
60
ROUGE-L
ROUGE-L
38
36
56
34
2
3
5
6
9 10 12
1
2
3
8 9 10 11 12 Time (future)
Time (future)
Oracle (40.2410.95)
Average (37)2110.51)
Oracle (59. (99.7310.76)
Average (60.05 Downscaling (50.1310.6)
Taylar (37.5111.33)
Downscaling (38.45.0.93)
Taylor (57 (57.85.1.13)
Recent 0.900
Recent (59.73+0.79)
(b) Average and Worst-case ROUGE-L for T5-small model. The left panel shows results for language
right panel news summarization. Each method is evaluated over future steps axis), with higher ROUGE-L values (y-axis) indicating better performance.
modeling
Figure F.12: Performance evaluation with perplexity and ROUGE-L of the T5-small model using various methods on language modeling and news summarization tasks.
Yearly results with T5-small model. We further evaluated interpolation and extrapolation strategies over two future years on the news summarization task using T5-small model. Consistent with prior monthly results, the Taylor expansion yielded weaker performance, with a significantly higher average perplexity of 9.274 + 0.060 compared to using the most recent model (average perplexity 7.582 + 0.019). Downscaling the parameters of the recent model achieved comparable average perplexity of 7.562 = 0.006. For the second future year, this scaling approach resulted in lower perplexity of 7.589 = 0,006 for downscaling compared 7.661 + 0.022 for the recent model. These findings from multi-year evaluation highlight the limitations of simpler Taylor series-based methods in this context and suggest that appropriate scaling of recent model parameters can be a simple strategy for temporal generalization.
Additional dimensionality reduction visualizations. We show TSNE visualization during lan- guage modeling and news summarization with continual learning in Figure F.17 and Figure F.18. We further show UMAP, PCA visualization during language modeling Figure F.16 and Figure F.19 and news summarization with continual learning in Figure F.20.
21

---

Published as a conference paper at ICLR 2026
Yearbook
5.0
Yearbook
Yearbook
20
2000
2.5
2000
2.5
PC2
0.0
1980
Year
t-NE 2
UMAP2
1980
-2.5
1960
-2.5
-5.0
1940
-7.5
-10
1940
10
PC
0
20
t-SNE 1
UMAP
Huffpostt
Huffpost
2017
Huffpost
2017
2017
2016
30
2016
2016
9.0
PC 2
2015
Year
L-SNE 2
20
2015
UMAP2
8.5
2015
Tear
2014
201
10
8.0
-2
2013
2013
7.5
2013
2012
2012
PC
-20
0
33
34
35
t-SNE
UMAP
Arxiv
Arxiv
2020
100
2020.0
6
2020.0
50
2017.3
2017.5
PC2
2015
Year
tSSE 2
2015.0
UMAP
2015.0
-50
2012.5
2012.5
2010
-100
2010.0
2010.0
10
-150
2007.3
2007.5
PC
100
20.0
22.5
25.0
L-SNE
UMAP
Fmow
Fmow
Fmow
5.0
100
12.5
12
12.5
PC2
2.5
0.0
10
Year
E-NNE 2
50
7.5
10.0
Your
UMAP2 2
10
10.0
7.5
5.0
-30
2.5
-5.0
10
-100
-100
0.0
5.0
2.5
0.0
t-SN
UMAP
Figure F.13: Dimensionality reduction of model parameters over time for WILDS-Time datasets. Each scatter plot shows 2D projection of model checkpoints using PCA, TSNE and UMAP dimensionality reduction techniques.
22

---

Published as a conference paper at ICLR 2026
encoder.bleck.5 Say. Sel.iaatett.ellltttttttiiioo SellAttention. weeggt
wttp:e/r.boo....lly....... www.eeeteeeetttniiinnn.n.o weightt
1.5
2016 09
1.5
2016 09
1.0
2016-01
2016-01
2015-09
201 5-15-09
PCA Component
2015-05
PCA Componant
2015-05
201 5-01
0,5
201 5-01
2014 09
2014-09
2014 05
2014.00 05
2014.01
2014.01
0.5
201 5-09
-0.5
201 5:09
2013:05
2013:05
2013-01
201 9-11
-1.0
2012 05
-15
2012 05
2012:01
PCA Component 1
PCA Component 1
encoder.block.s.kk httpe/rcoooosllleeeeeeeeeeeeeees wi.weght
srrrcerrrboo....llyeeeeeeeeeeeee wi.
20:6-09
2016-09
PCA Component 2orada
20:6-01
2016-01
2015-09
2015-09
PCA Component
1
20:5-01
2015-01
2014-09
5
5
2014.05
2014.05
2
-1-
20:3-009
2013-09
-2-
2013:01
2013:01
2012:09
2012:05
2012:05
2012:01
2
PCA Companent
PCA Companent
encoder.block.5.layy Senoodrrbbocks.llyey.eeeeeeeeeee www.eeeeeeeeeeeeeeeee....... wwtp://weeeeeewwiiiiiil..
decoder.biockk.. 5.ayerr22 Dew.eeeeeDeeeeeeeeeeee.... wo, weight
2016-09
2016-09
2016 05
2016 05
2016:01
2016
20:5-09
2015-09
PCA Component
PCA Component
20:5-01
1
2015-01
2014 05
2014 05
01
01
2013:09
1
2013-09
05
2013:01
2013-01
2012-09
2012:05
-3
2012:05
-6
-2
0
2
2012 01
2
2
2012 01
PCA Component 1
PCA Component 1
encoder.final_aaye Jlayer_nomm wwr.noooommhhhhhhe
decoder.fina http:/ffi__alyyeeooommweaeee wwwrr/ooowwweteee weight
2016-09
2036-09
2036 05
2036 os
2016-01
2016-01
2015-09
2015-09
PCA Companent
PCA 2omponent2
2015 01
2015 01
2014-09
aes.
2014 05
000 a o0ap pdo
2014 05
2014-01
2013-09
2013-09
2013-05
-0.05
2013-05
2013
0.10
2012 09
-0.10
2032 09
2012
2012 01
2032.00
-0.2
-0.1
0.0
0.2
PCA Component
PCA Component
Figure F.1:: Smooth parameter trajectories under continual learning. PCA of T5-small weights during news summarization with continual learning shows smooth, coherent evolution over time. This contrasts with independently fine-tuned checkpoints (Figure F.15), highlighting how continual learning preserves temporal consistency in parameter space and supports forward transfer.
23

---

Published as a conference paper at ICLR 2026
encoder.bleck:S..e http://ce.S.Saaallleelleeeeenoon welllAtteenioo..weiighh
decoode.blockk... htwp:e/eeooc.Slllee....eeeeettte
0.6
2016 09
2016 09
2016-01
0.4
2016-01
0.4
2015-09
201 5-09
PCA Component
2015-05
PCA Component
0.2
00
2015-05
2015-01
201
2014 09
2014:00 09
2014 05
0.0
2014.00 05
2014.01
0.0
201 5-09
201 5:09
2013:05
0.2
2013:05
2013-01
2013-01
-0.2
0.4
2012 05
2012 05
-0.2
0.0
0.2
0.4
-0.4
-0.2
0.0
0.2
0.4
PCA Component 1
PCA Component 1
encoder.blockk5.pg wtw.aayeee..eeeeeelleeeeeeee wi. weight
Secooer.blockk5.ayyeee 2 DenseReluDense wi weight
2018-09
2016-09
1.0
2016-01
2016-03
2015-09
2035
PCA Component
0.5
PCA Companent
2015-01
2015-01
2014-09
2014.05
0.00
2014-05
2014-03
20133009
-0.25
2013-09
2013:01
-0.50
2013-01
2012-09
2012:09
2012:05
-0.75
2012 05
0.5
0.4
0.2
0.0
0.2
0.4
0.6
o.s
2012:01
0.75
0.50
0.25
0.00
0:25
0.50
0.75
2032-03
PCA Component
PCA Companent
weight
decoder, wew.aayerr2Derreeeeeeeeeeeeeeee we.wightt
2016-09
0.8
2018-09
2016.05
2016 05
1.0
2016.01
2016.02
2015-09
2015-09
PCA Component
PCA Component
2015-01
0,2
2015-01
201. 05
0,0
2016 05
2013:09
2013:09
0.5
2013:01
0
2013:01
2012:09
-0.6
2012:09
-1.0
2012 01
2012:05
0.8
2012:05
2012 01
0.6
-0.4
-0.2
0.0
0.2
0.4
0.6
-0.75
-050
-025
0.00
0.25
0.50 0.75
PCA Component
PCA Component
encoder.final Laybeerrrinal_ayeerrrre weight
decoder.final Llaadeeerfnallllyerrreeeee weight
2016-09
2036-09
2036 05
0.06
2036 05
2016-01
2036-01
2015-09
0.04
2015-09
PCA Companert
PCA Component2
2015 01
0.02
2015 01
2014-09
2014-09
2014 05
0.00
2014 05
2014-01
2014-02
0.80
2013-09
0.02
2013-09
2013-05
2013-05
0.02
0.04
-0.04
2012 09
2012.05
-0.06
2002 09
2012.01
2032.01
-0.050-0.022
0.000 0.025 0.050 0.075 0.000
-0.075-0.590-00022 0.000 0.025 0.050 0.075
PCA Component
PCA Component
Figure F.15: Parameter trajectories from independent finetuning, PCA of TS-small weights dur- ing summarization, where each timestamp model is trained independently from pre-trained checkpoint. The lack of smoothness across time steps indicates inconsistent parameter evolution and highlights the challenges of extrapolation in the absence of continual learning.
24

---

Published as a conference paper at ICLR 2026
encoder.block. Salyee. wwl.Attention..w..t.. w.weihtt
2016-11
20
2016-11
2016-08
2016-08
15-
10
10
PCA Companent?
PCA Comparant
2014-04
2015-10
2013-10
2013-07
2013-07
2013-06
2033-04
2013-01
2013-01
2012-10
-10
2012-10
2012-05
2012-06
2012-03
2012:00
-15
2012-03
2012-00
Component
Component
encoder.block. Saa.:oe.sseayeeeeeeeeeeeeeeeeee weight
decoder. bbr. weight
2016 11
40
2016 11
30
2016 08
2016 08
2016-04
2016-04
2016-01
2016-11
07
PCA Component
PCA Component
2013-07
2013-04
-10
2013-04
20
2013:01
10
2013:01
10
06
-30
PCA Component
PCA Component
encoder www.eeeneeeeeeeeeeee.......
decoder.block.5 Dew.eeeeeeeeeeeeeeeeeeeeeee
2016-11
2016-11
30
2016 08
2016
2016 08
2016 04
2016-01
2016
2015-10
2015-10
2015-07
2015-07
PCA Compunent
2015-04
PCA Comgenent
2015-04
2014-10
10
2014-10
2014-07
2015-04
-10
2013-04
-20
2013-01
2012-10
-20
2013-01
2012-10
2012
06
PCA Component
PCN Component
encoder.final layer, norm.weight
decoder.final layer_norm. wwrrnnnoooweeeeeeeee worm.weight
2016-11
2016-11
2016-04
2016-08
0.8
2016-08
2016-04
2016-01
2016-01
2015 10
2015
2015
0.6
2015-07
PCA Companent
2015-04
PCA Component
2015-0A
2015-01
2015-01
2014-19
2014-10
2014-07
2014-07
2014-04
Ruggeaog Bad 88
2014-04
G ege
07
0,0
-0.04
-02
-2
PCA Component
PCA Component
Figure F.16: Smooth parameter trajectories under continual learning. PCA of T5-small weights
during language modeling with continual learning shows smooth, coherent evolution over time.
25

---

Published as a conference paper at ICLR 2026
encoder.block. 5.layer wwwfftteettennnnn.......o weightt (TSNE)
(TSN))
2016-11
2016-11
2016-08
2016-08
2016-04
2016:02
2016
2015
1-
2015
2015
2015 04
2015
2015-01
2015-01
Dkn 2
20144-10
2014407
Din
2014-10
2014-07
2014-04
2014-04
2014-01
2014-01
20:3-10
2013-30
20:3:07
2013:47
-3-
20:3:04
2013 04
2023
2013 01
2012 10
2012 30
0012-06
2012-06
0.0pp
20:2-03
00
2012-68
20:2:00
2012:00
-15
-1.0
-0.5
0.0
1.5
-12
0005 -0.4
Dim
Dim
kooder.block...lag DeweerrdDesseeeluDenneeee weight (TSNE)
Recoderblook.5.layy.. Dew.eree..essseelueeeeeeeeeeee wi. weight (TSNE)
2010
08
1.0
2015-04
2015-01
2014-10
Dim
2014-07
Dim
2016-07
2014-04
2014-04
2014-0:
2014-01
20:3-07
20:3-10
-0.5
2013-10
2013-07
20:3:04
2013:04
20:3:01
2013:01
20:2:10
-1.0
2012
2012 06
2012 06
90
2012 03
2012 03
2012:00
2012-00
-2.2
-2.0
Dim
htw.blll....lllll...eeeeeeeeeeed we.ighh (TSNE)
httprr/lccc....ae....eeeee weight
Dim
10
2014.07
Dim
2014-07 E
2014.04
2014.01
2013
2013
2013-01
8-
2012-10
2012-10
2012-06
2012-06
2012
2012 03
2012 00
2012 00
Dim
encoder.final layer weight (TSNE)
decoder.final layer_norm..eeeeee norm.weight (TSNE)
2.0
2016-11
2016-11
2016-08
1.5
2016-04
2016-04
2016-01
So even oo ou 0 00
1.0
2015-07
2
0.5
Dim
0.0
Dim2 2
2014-07 <
2014-04
2014-04
g
-0.5
2014.01
2013:10
2014.01
04
2013
-1.
2012-10
8 8oo
2012:
2012-06
2012-06
2012-03
2012-03
2012-00
2012-00
Figure F.17: TSNE of T5-small weights during language modeling with continual learning.
26

---

Published as a conference paper at ICLR 2026
encoder.block..layye wtw.llayrr..SSfftteetttiiio Selffttention....eig.. (TSNE)
5 htw.s/aaa....neeeeeeeeeet.oo.oo w.w.egt (TSNE)
2016-09
2016-05
2016-05
2016-01
2015-69
2015-09
12
2015 01
2015 01
Dim 2
11
Dim
2014-05
2014-05
2013-09
2013-09
2013-05
2013-005
2012:09
2012-09
600
2012:01
2012:01
1.5
2.5
3.5
Dim
Dim
1coder.block.5.ppg w.w.kk...aayr...... Den...eeeeeeeeeeseee.e... wwinnse.w..wlgg... weigh (TSNE)
bcoder.block...layy.. w.w.a......eeeeeeeee... www.eeeeeeeeeeeeeeeeeeee DDense wi weight (TSNE)
2016-05
2016-05
2015
2015
2015:01
2015-01 01
Dim
Dim
2014-05
2014-05 0
2014:01
2016-01
20:3-05
2013-05
20:2-09
2012-09
-4
00000
2012:01
2012:01
-22
-2.0
-1.2 -1.0
0.0
k.oder.blcck....o w.w.ee...ly......e.... www.:eeeeeeDDDeeeeee..eeeee (TSNE)
coder.block.5. wttp:ok.....e..eeeeee..ee www.ayyee2..eeeeeeeeeeeeeeeeeeee (TSNE)
od
2016
2016.05
2016-05
2015:09
11
2015-09
2015-01
2015-01
Dimn
2014 09
2014:05
Dim
2014 09
2014:05 8
2014 01
2014 01
-2
20:3-05
2013-05
2012 09
2012 09
-4-
6
2012 01
2012.05
6-
2012 01
0.0
encoder.final layer weight (TSNE)
deroder.final Jaayaaaaaaaaaaa weight (TSNE)
d GOO Buslicug 0000 00000 aoo
5-
2016-05
-0.23
2016-05
2016-0:
2016-03
2015:05
-0.75
2015 05
Dim
2014.02
Dim
-1.00
2024 09
2014-05
2014-01
-1.25
2034-03
1.50
2013 05
2013 05
-1-
2012
2012 09
-2.00
2612-01
2012-01
Figure F.18: TSNE of T5-small weights during news summarization with continual learning.
27

---

Published as a conference paper at ICLR 2026
srtooerblockk.llyyrooeeeffae w.weight (UMAR)
icoder.block. wwwerel.eneecttennnnn...... (UMAP)
2016-11
20:6-11
20.0
2016-08
2016-08
2016-04
2016-04
17.5
2016
10
2015
2016
and gomn dire
2015
2015
2015-01
2015-01
2010-10
2014-10
Dkn
2014-07
Din
2014-07
2014-04
2014-04
2014-01
2014-01
2013-10
2013-30
2013:07
2013:47
2013:04
2013 04
2013 01
2013 01
2012 10
2012 30
2012 06
2012-00
2012-03
2012-68
2012:00
2012:00
7.5
15.0
17.5
12
Dim
Dim
icoder.block.. 1 wew.eeeeeeeeReeeeeeeeee.e wwwp://ennwwwwi.iiii..e.... weight
20.6
Ecoder.block. De weight (UMAP)
2016
5.5
2015-01
Dim
2014-07
DIm
2014-07 JG
20144-44
2014-04 0
2014-01
2014-01
20:3-10
2013-10
20:3-07
2
2013-07
20:3-04
2013:04
20:3:01
2013:01
2012-10
2012
2012 06
2012 06
15
2012 03
2012-00
-2
10
2012
2012:00
coder block...kkkkkkko www.:/laay.r....c. Denseeeeeeeeeeeeeeeeeee..e.. we.weggtt (UMA))
coder.block. wawee... www.DeeeeeeDDeee....... weight 20160
22
2-
18
Dim
Dim
2014.07 E
2014:04
2014.04
2014.01
2014.01
14
2013
2013
2013:01
2013-01
2012-19
2012-06
3
2012-10
2012-06
20
2012 03
2012 00
2012 00
10
Dim
encoder.final layer weight (UMAP)
decaderfinal layer weight (UMAP)
2016-11
2016-11
13
2016-08
2016-04
10.0
2016-01
12
2015-10
7.5
11
2014-10
Dim
10
Dim
2.5
2014-04
2014.01
2013-10
07
-2.5
04
2012:10
-5.0
2012-10
2012-06
2012-06
2012-03
2012-03
2012-00
2012-00
Figure F.19: UMAP of T5-small weights during language modeling with continual learning.
28

---

Published as a conference paper at ICLR 2026
ancoder.blockk..layerr wwwp:e/.SSSeetteentoon.ooooo (UMAP)
acoder.loock..layy www../yeeeeeeeeeeeeeeeoooooooooo (UMAP)
2016:09
2016-05
10
2016-05
2016:01
2016:01
2015-69
2015-09
600
2015 01
2015 01
5
Din
2014-05
8
Din
2014-05 8
%
2013-09
2013-09
2013-05
0013-03
2012:09
2012:09
%
2012:01
2012:01
100.0
-7.
-5.0
0.0
2.5
Dim
Dim
coder 5.a.err. Dewerreeeeeeeeeeeeeeeeee.e wi. wiight (UMAP)
:coder. www.bboookkk..y.e... w.w.ea....eeeeeellieeeee.. weight (UMAP)
2016-05
2016-05
2015:09
2015
2015 01
2015 01
Dim
Dm
2014-05
2014-05
2014
2016-01
8-
2013-05
2013-05
2012:09
2012-09
2012 01
2012:01
7.5
coder block...laapp.... b.w.ck................. Bea.eee...enseeeluuuee wo.weiggtt (UMAP)
coderr 5.ock..boock..ko 2aw.kk...aayr.2...3 Deyee:.DenseRelueeeeee.... weight (UMAP)
24 0 Ragi
2016:05
2016-05
2015:09
2015-09
20
ao Boll & agen on
2015-01
2015-01
Dim
18
2014 09
Dim
2014 09
2014:05
2014 01
2014 01
14
2013-05
2013-05
2012 09
2012 09
2012 01
2012 01
-5.0
0.0
12.5
encoder.finl layer_ weight (UMAP)
decader.final layer weight (UMAP)
18
2016-05
-1.0
2016-05
2016-01
16
2015:09
-1.5
2015-09 09
2015:05
2015 05
2015 01
Dim
00
2014.03 09
Dim
2014.00
2014-01
-.5
2014-01
2013:00
2013 05
2013 05
0
2012
-3.5
2012 09
2012-01
2012-01
Figure F.20: UMAP of Tm-small weights during news summarization with continual learning.
29


Reviews:
[{"review_id": "x69Icl1AxS", "reviewer": "Reviewer_Zdv8", "summary": "Due to inaccurate prediction on unseen future data by Machine Learning (ML) models, the authors investigate under what conditions models can achieve such a generalization when depending on past data. The authors explore two Primary approaches: Parameter Interpolation and Parameter Extrapolation. Furthermore, several methods were benchmarked on a set of temporal tasks, including language modeling, news summarization, tag prediction, academic paper categorization, and land use classification.", "questions": "1. What was the data size for each time stamp when using different datasets for different methods? \n2. While the paper includes an Oracle model in Figures 2, its definition and implementation are not clearly explained in the main text.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It presents a large-scale, systematic evaluation across multiple tasks (language modeling, summarization, paper categorization, land use classification, and photo gender prediction) and datasets (NewsRoom, Wilds-Time, arXiv, HuffPost, FMoW, Yearbook).\n\n2. The paper makes a contribution by showing that none of the proposed or existing methods reliably outperform a simple \"most recent model\" baseline.\n3. Through parameter norm analysis and dimensionality reduction (PCA/UMAP), the paper provides intuitive interpretations of why downscaling helps.\n4. The authors emphasize reproducibility and maintaining transparent reporting of hyperparameters and evaluation setups.", "weaknesses": "1. The paper primarily evaluates existing approaches (parameter interpolation, extrapolation, downscaling) rather than introducing a new algorithm or theoretical framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S9rBto3IYi", "reviewer": "Reviewer_KbVf", "summary": "The paper studies temporal generalization — the ability of models trained on past data (and only past data) to perform well on future, potentially shifted data. The authors point out that many recent papers implicitly assume that, given a sequence of checkpoints, one can interpolate or extrapolate those weights to obtain parameters that work better on future data, even when the future distribution is unknown. The central claim of the paper is negative but important: under a realistic, “no access to the future” setting with streaming / continual updates, none of the evaluated interpolation (uniform / EMA merging, recent model, norm-downscaling) or extrapolation (first-order Taylor on parameter trajectory) methods consistently beats the trivial baseline of “just use the latest model. The only method that consistently avoids hurting performance is a simple downscaling of the latest model (interpolating toward the origin), which the authors motivate via parameter-norm growth and sharper minima over time.", "questions": "1. You show that parameter norms steadily increase under CL and that downscaling helps. Did you try layerwise or modulewise downscaling (e.g. scale only attention / only FFN / only classifier) and, if so, was the benefit still present? Intuitively, temporal brittleness might concentrate in the last layers that overfit to very recent tokens / labels.\n2. Can you tell whether the sign flip correlates with which dataset was seen last (e.g. news vs scientific vs vision)? In other words, is the problem that parameter trajectories are globally noisy, or that mixing modalities / tasks breaks linearity?\n3. Fig. 6 clearly shows CL helps. But you fine-tune from θt−1 on all datasets. What happens if the per-step compute is reduced (e.g. 10% of the usual steps, or LoRA-style adapters only)? Do interpolation/extrapolation fail immediately because checkpoints become less aligned, or do they degrade more gracefully?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is very explicit about the setting: streaming / continual fine-tuning, no access to future unlabeled or labeled data, and hyperparameters must be chosen from the past. This closes an often-overlooked loophole in earlier temporal/DG works that quietly validated on future data.\n2. The experiments span: (i) monthly news language modeling and summarization with T5 (showing clear temporal decay and the salience of downscaling), and (ii) WILDS-Time datasets (Yearbook, HuffPost, arXiv, FMoW), where no single method dominates. This makes the main negative result much more convincing than if it were shown on one dataset.\n3. The paper separates parameter interpolation (merging, recent, downscaling) from parameter extrapolation (Taylor step using the most recent difference).\n4. The observation that the optimal extrapolation factor 𝛼 often drifts below 1 or even becomes negative (meaning “don’t extrapolate; sometimes go backward”) is a strong piece of evidence that parameter trajectories of large models are not sufficiently smooth or aligned over time to support simple Taylor-based prediction.\n5. The finding that downscaling the latest model is the only option that does not reliably hurt, and that this correlates with steadily growing parameter norms under continual learning, is a small but real contribution that practitioners can immediately try.", "weaknesses": "1. The paper explains why interpolation/extrapolation is hard via non-identifiability, non-convexity, and noisy parameter trajectories, but these are largely qualitative or synthetic-illustration arguments; there is no general impossibility statement of the form “under assumptions X, any future-agnostic checkpoint-only method cannot beat the latest model by more than 𝜖”. A lightweight formalization, even under a stylized model, would sharpen the negative result.\n2. The paper itself shows that if you don’t do sequential fine-tuning (i.e. if you reinit from a pretrain every time), all methods degrade much more (Fig. 6). That means the key negative result is partly conditional on the positive result that CL keeps consecutive checkpoints close. This is important, but it also means the conclusions are strongest for CL-like pipelines, weaker for other update regimes.\n3. The paper does the right thing — tune 𝛼 only on past data — but for some methods (esp. extrapolation) the chosen 𝛼 fluctuates sharply over time (Fig. 5). In practice, this means users would have to re-tune at every step, which undercuts the appeal of “cheap temporal generalization.” A small ablation on “fixed  𝛼 over long horizons” vs “retuned 𝛼 ” would have made this clearer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tAdFnmeJET", "reviewer": "Reviewer_z2gD", "summary": "This study asks whether ML models can truly generalize to future, unseen data when you only have access to past model checkpoints—no peeking at future data. The authors benchmark two families of methods that operate purely on historical parameters: (i) parameter interpolation (averaging/EMA and a simple downscaling of the latest model), and (ii) parameter extrapolation (first-order, Taylor-style forecasting of weights). Across language modeling and summarization on monthly news, plus yearly tasks like Yearbook, HuffPost, arXiv, and FMoW, they find no approach reliably beats simply deploying the most recent model; notably, downscaling the latest weights is the only method that doesn’t consistently degrade performance.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper introduces a highly realistic experimental framework that mirrors real-world deployment, strictly prohibiting the use of future data.\n\nIt delivers a comprehensive empirical evaluation, testing multiple architectures and datasets with different temporal frequencies and scales.\n\nThe authors offer transparent and reproducible experimentation, including open-source code and detailed implementation settings.\n\nThe study provides valuable negative evidence, demonstrating that intuitive weight-based strategies can fail dramatically, which grounds future work in reality rather than optimism.\n\nBeyond reporting results, it also diagnoses the causes—linking observed failures to theoretical properties such as non-identifiability and parameter-space discontinuity.", "weaknesses": "The investigation remains restricted to parameter-space manipulation, leaving out more adaptive or data-driven temporal methods that could exploit unlabeled future samples.\n\nSome evaluated datasets exhibit limited temporal granularity, reducing the precision with which the authors can analyze temporal drift.\n\nThe reliance on historical validation only makes hyperparameter selection brittle and may underestimate the potential of some methods.\n\nWhile empirically thorough, the paper’s methodological contribution is modest, focusing on benchmarking rather than innovation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JNibZ1TRnl", "reviewer": "Reviewer_azAb", "summary": "This paper deals with the issue of predictions on future data based on past checkpoints, presenting comprehensive benchmarking on various tasks (language modeling on evolving news corpora, news summarization and tag prediction, categorization\nof academic papers, land use classification from satellite imagery reflecting changes over several\nyears, and gender prediction from historical yearbook photos ). Existing benchmarking studies are hampered by unrealistic evaluation settings as they e.g. assume access to future data.\nThe core hypothesis is that approaches can be broken down into two phenomena: interpolation vs. extrapolation. They adopt a sequential fine-tuning approach.\n\nTheir core findings are:\n1) No method consistently improves performance over the recent model.\n2) Taylor-Series extrapolation underperforms compared to other methods.\n3) Continual learning is important.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The experiments presented in the paper are extensive and are clearly guided by research questions, which are answered and discussed well.", "weaknesses": "There are minor parts of the presentation that could be improved, e.g. spell-checking, using the same font for all figures and the same style for all figure captions. Table 1 is also positioned strangely.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)