Continuous Temporal Domain Generalization
Authors: Zekun Cai
Guangji Bai
Renhe Jiang
Xuan Song
Liang Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: Domain Generalization
Temporal Domain Generalization
Continuous Dynamics
Koopman Operator
Concept Drift
Neural ODEs
Primary area: optimization_for_deep_networks
TL;DR: This work defines Continuous Temporal Domain Generalization. By leveraging a Koopman operator-driven continuous system, the paper effectively captures the essence of temporal generalization through the synchronization of model and data dynamics.

Abstract: Temporal Domain Generalization (TDG) addresses the challenge of training predictive models under temporally varying data distributions. Traditional TDG approaches typically focus on domain data collected at fixed, discrete time intervals, which limits their capability to capture the inherent dynamics within continuous-evolving and irregularly-observed temporal domains. To overcome this, this work formalizes the concept of Continuous Temporal Domain Generalization (CTDG), where domain data are derived from continuous times and are collected at arbitrary times. CTDG tackles critical challenges including: 1) Characterizing the continuous dynamics of both data and models, 2) Learning complex high-dimensional nonlinear dynamics, and 3) Optimizing and controlling the generalization across continuous temporal domains. To address them, we propose a Koopman operator-driven continuous temporal domain generalization (Koodos) framework.  We formulate the problem within a continuous dynamic system and leverage the Koopman theory to learn the underlying dynamics; the framework is further enhanced with a comprehensive optimization strategy equipped with analysis and control driven by prior knowledge of the dynamics patterns. Extensive experiments demonstrate the effectiveness and efficiency of our approach. The code can be found at: https://github.com/Zekun-Cai/Koodos.

Continuous Temporal Domain Generalization
Zekun Cail Guangji Bai², Renhe Jiang", Xuan Song and Liang Zhao?
The University of Tokyo, Tokyo, Japan
Emory University, Atlanta, GA, USA JJilin University, Changchun, China
4 "Sothhern University of Science and Technology. Shenzhen, China {caizekun, jiangrh, songxuan @csiss u-tokyo.ac. {guangj i. bai liang. zhao Cemory. edu
Abstract
Temporal Domain Generalization (TDG) addresses the challenge of training pre- dictive models under temporally varying data distributions. Traditional TDG approaches typically focus on domain data collected fixed, discrete time in- tervals, which limits their capability to capture the inherent dynamics within continuous-evolving and rregularly-observee temporal domains. To overcome this, this work formalizes the concept of Continuous Temporal Domain General- ization (CTDG), where domain data are derived from continuous times and are collected at arbitrary times. CTDG tackles critical challenges including: I) Charac- terizing the continuous dynamics of both data and models, 2) Learning complex high-dimensional nonlinear dynamics, and 3) Optimizing and controlling the gen- eralization across continuous temporal domains. To address them, we propose a Koopman pverator-drieer continuous temporal domain generalization (Koodos) framework. We formulate the problem within a continuous dynamic system and leverage the Koopman theory learn the underlying dynamies; the framework is further enhanced with a comprehensive optimization strategy equipped with analysis and control driven by prior knowledge of the dynamics patterns. Extensive experiments demonstrate the effectiveness and efficiency of our approach. The code can be found at: https://github. com Caii-KoCooKKK
Introduction
In practice, the distribution of training data often differs from that of test data, leading failure in generalizing models outside their training environments. Domain Generalization (DG) 49 46
14 40
machine learning strategy designed learn generalized model that performs well on the unseen target domain. The task becomes particularly pronounced in dynamic environments
where the statistical properties of the target domains change time 16
33
4 prompting the
development of Temporal Domain Generalization (TDG) (27 39: 41 2 55 57
TDG recognizes
that domain shifts are temporally correlated. extends DG approaches by modeling domains as sequence rather than as categorical entities, making it especially beneficial in fields where data is inherently varying.
Existing works in TDG typically concentrate the discrete temporal domain, where domains defined by distinct "points in time" with fixed time intervals, such as second-by-sccom data (Rot 2-Moons 48) and annual data (Yearbook (54)). In this framework, data bounded in time interval are considered as a detached domain, and TDG approaches primarily employ probabilistic models to predict domain evolutions. For example, LSSAE 41 employs a probabilistic generative model to analyze latent structures within domains; DRAIN 2 builds a Bayesian framework to predict future model parameters, and TKNets [57] constructs domain transition matrix derived from the data.
Corresponding author 38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Temporal Test Domains
Temporal Training Domain on Continaous Time
May Appear at Anylime with Model State ?
Tweets aas Red
Tweets asa liue Tweets bbbb Red Tweets
hal:: Opinice
Twerts lob skue
Tweets Blue
Rosdts
Roshs
Roodo
Results
??
Chastter
Data Cooditer
follwing Gurder 3/7
Eeche Oeat
fag. lowing
Bonaln
Twoets
Doman
Dernats Dats
D
Domain Dats
Damals Data
twezta 12:00
203
Dats
7:3
Steam Data
*
08-31
24-01-25 23
24-02-02
03
A 24-04-04
24-05
Tlma
12:23:12
05:30:45
10:43:32
23:30:12
12:32:32
13:25:23
Anytime Anytimo
Figure I: An example of continuous temporal domain generalization. Consider training classification models for public opinion prediction via tweets, where the training domains are only available at specific political events (e.g.. presidential debates), we wish to generalize the model to any future based on the underlying data distribution drift within the time-irregularly distributed training domains.
However, in practice, data may not always occur or be observed at discrete, regularly spaced time points. Instead, events and observations unfold irregularly and unpredictably in the time dimension, leading to temporal domains distributed irregularly and sparsely over continuous time. Formally, this paper introduces a problem termed Continuous Temporal Domain Generalization (CTDG), where both seen and unseen tasks reside in different domains at continuous, irregular time points. Fig. [ illustrates example of public opinion prediction after political events via Twitter data. Unlike the assumption in traditional TDG that the temporal regularly domains, the data are collected for the times near political events that may occur in arbitrary times. In the meanwhile, the domain data evolve constantly and continuously over time, e.g., active users increase, new friendships are formed, and age and gender distribution changes. Correspondingly, the ideal classifier should gradually change with the domain at random moments to counter the data distribution change over time. Finally, we concerned with the state of the predictive model at any moment in the future. CTDG is ubiquitous in other fields. For example, in disaster management, relevant data are collected during and following disasters, which may occur at any time throughout the year. In healthcare, critical information about diagnosis and treatment is typically only documented during episodes of care rather than evenly throughout the lifetime. Hence, the CTDG task necessitates the characterization of the continuous dynamics of irregular dintriiitt domains, which cannot be handled by existing TDG methods designed for discrete-dynamics and fixed-interval times.
Despite the importance of CTDG, it is still highly open research area that is not well explored because of several critical hurdles in: 1) Characterizing data dynamics and their impact on model dynamics. The irregular times of the temporal domains require us to characterize the continuous dynamics of the data and, hence, the model dynamics ultimately. However, the continuous-ümme data dynamics are unknown and need to be learned across arbitrary time points. Furthermore, is imperative yet challenging to know how the model evolves according to the data dynamics in continuous times. Therefore, we don't have a direct observation of the data dynamics and the model dynamics we want to learn, which prevents us from existing continuous time modeling techniques. 2) Learning the underlying dynamies of prempparametrieed models. Deep neural networks (e.g.. Multi-Layye Perceptron and Convolutional Neural Network) are highly nonlinear and over-parametrizeer and hence, the evolutionary dynamics of model states over continuous time are high-dimensional and nonlinear. Consequently, the principal dynamics reside in a great number of latent dimensions. Properly representing and mapping these dynamics into a learnable space remains a challenge. 3) Jointly optimizing the model and its dynamics under possible inductive bias. The model learning for individual domains will be entangled with the learning of the continuous dynamics across these models. Furthermore, in many situations, we may have some high-level prior knowledge about the dynamics, such as whether there are convergent, divergent, or periodic patterns. It is an important yet open topic to embed them into the CTDG problem-solving
To address all the challenges, we propose the Koopman operator-driven continuous temporal domain generalization framework (Koodos). Specifically, the Koodos framework articulates the evolutionary continuity of the predictive models in CTDG, then leverages continuous dynamic approach model its smooth evolution over time. Koodos further simplifies the nonlinear model system by projecting them into linearized space via the Koopman Theory. Finally, Koodos provides interface that
2

---

reveals the internal model dynamic characteristies, as well as incorporates prior knowledge and constraints directly into the joint learning process.
2 Related works
Domain Generalization (DG) and Domain Adaptation (DA). DG approaches attempt learn
a model from multiple source domains that generalize well to an unseen domain [38 37 28] 5
13 56]_
Existing DG methods can be classified into three strategies 44 (1) Data manipulation
techniques, such as data augmentation [45 46 57 and data generation [32 40 (2) Representation
learning focuses on extracting Nomain-invariian features [7 and disentangling
-ohain--hared
from domain-specifer features [30] (3) Learning strategies encompass ensemble learning [34],
meta-learning 26 14 9, and gradient-based approaches [2T]. Unlike DG, DA methods require
simultaneously accessing source and target domain data to facilitate alignment and adaptation
[50
51 29]. The technique includes domain-invann domain-invariant invvarint learning [17] (7] [35] 48] [53], domain mapping
6
20 15 31, ensemble methods [43). and so on. Both DG and DA are limited to considering
generalization across categorical domains, which treats domains as individuals but ignores the smooth evolution of them over time.
Temporal Domain Generalization (TDG). TDG is an emerging field that extends traditional DG techniques to address challenges associated with time-varying data distributions. TDG decouples time from the domain and constructs domain sequences to capture its evolutionary relationships. S-MLDG
27 pioneers sequential domain DG framework based on meta-learning. Gradient Interpolation (GI)
proposes to extrapolate the generalized model by supervising the first-order Taylor expansion of the learned function. LSSAE [41] deploys a probabilistic framework to explore the underlying structure in the latent space of predictive models. DRAIN [2] constructs a recurrent neural network that dynamically generates model parameters to adapt to changing domains. TKNets (57) minimize the divergence between forecasted and actual domain data distributions capture temporal patterns.
39
Despite these studies, traditional TDG methods are limited to requiring domains presented in discrete time, which disrupts the inherent continuity of changes in data distribution, and the generalization can only be carried forward by very limited steps. No work treats time as continuous variable, thereby failing to capture the full dynamics of evolving domains and generalize to any moment in the future.
Continuous Dynamical Systems (CDS). CDS are fundamental in understanding how systems evolve without the constraints of discrete intervals. They are the study of the dynamics for systems defined by differential equations. The linear multistep method or the Runge- Kutta method can solve the Order Differential Equations (ODEs) [19]. Distinguishing from traditional methods, Neural ODEs I represent a significant advancement in the field of CDS. It defines a hidden state as a solution to the ODEs initial-value problem, and parameterizes the derivatives of the hidden state using neural network. The hidden state can then be evaluated at any desired time using a numerical ODEs solver. Many recent studies have proposed on which to learn differential equations from data 42
23 36)
3 Problem definition
Continuous Temporal Domain Generalization (CTDG): We address prediction tasks where the data distribution evolves over time. In predictive modeling, a domain D(t) is defined as a dataset
collected at time / consisting of instances {ax{") (1) V; () E x((1) x (()) N()) where x;; : V;
(!) (t) and
(1) represent the feature, target and the number of instances at time L, and (1), (t) denote the input feature space and label space at time t, respectively. We focus on the existence of gradual concept drift across continuous time, indicating that domain conditional probability distributions P(Y (t),, with X (t) and Y (t) representing the random variables for features and targets at time t, change smoothly and seamlessly over continuous time like streams without abrupt jumps.
During training, we are provided with a sequence of observed domains (D(t)), (D(t)),D(t)) D(t2), D(tr)) collected at arbitrary times T = (2,,t,,.,, 17). where i; E R and t < 12 < < lr. For each V(4),, where
domain ((4:) at time 4 E T, we learn the predictive model g(: ((4)) x(4) 0(4) denotes the parameters of function y at timestamp 4, We model the dynamics across the parameters (0(t)), 0())),((t)))))))))) ......................... ,((r)) }. and finally predict the parameters 0(s) for the predictive model g(::0(s)) X(s) + (s) at any given time & TT For simplicity, in subsequent, will use D., X,, Y,, 0, to represent D(t,). X(t),. Y (t.). O(t.) at time tq.
Unlike traditional temporal domain generalization approaches (39 2 52
that divide time into discrete intervals and require domain data be collected at fixed time steps, the CTDG problem
3

---

KorpmunsSpued
Value frum <
90-1
intiia! Fafure
B2
Modiel Space
BT
Lecennt
D3
Muntri
Time ott is
f2
by Time
(a) Three Spaces and Parallel Flows in Continuous Temporal Domains
(b) State Constraints Given Moment
Figure 2: Macro-flowe flows and nicroo-onnstraints in the proposed model framework.
treats time continuous variable, allowing for the time points in training and test set to be any arbitrary positive real numbers, and requiring models to deal with the temporal domains as continuous streams. CTDG problem poses several unprecedented, substantial challenges in: 1) Characterizing the continuous dynamics of data and how that decides model dynamics; 2) Modeling the low-dimensional continuous dynamics of the predictive model, which embedded in high-dimensional space due to over parrpearamtti::tionn; and 3) Optimizing and analyzing the continuous predictive model system, including the application of inductive biases control its behavior.
4 Methodology
In this section, we outline our strategies tackle the problem of CTDG by overcoming substantial challenges. First, to learn the continuous drifts of data and models, synchronize between the evolving domain and the model by establishing the continuity of the model parameters over time, which is ensured through the formulation of differential equations elaborated in Section 4.1 To fill the gap between high-dimensional model parameters and low-dimensional model continuous dynamics, simplify the representation of complex dynamics into principal, well-characterized Koopman space as detailed in Section
To jointly learn the model and its dynamics under additional inductive bias, in Section 4.3] we design series of organized loss functions to form efficient end-to-end optimization strategy. Overall, as shown in Fig. a]a). there are three dynamic flows in our system, which the Data Flow, the Model Flow, and the Koopman Representation Flow. Through the proposed framework, we aim to ensure not only that the model responds to the statistical variations inherent in the data, but also that the characteristics of the three flows are consistent.
4.2
4.1 Characterizing the continuous dynamics of the data and the model
In this section, we explore the relationship between the evolving continuous domains and the corresponding dynamics of the predictive models. We demonstrate that the dynamics temporal domains lead to the internal update of the predictive model continuously over time in Theorem [ Following this, we develop a learnable dynamic system for the continuous state of the model by synchronizing the behaviors of the domain and model.
Assumption 1. Consider the gradual concept drift within the continuous temporal domains. It is assumed that the conditional probability distribution P:(Y X) changes continuously over time, and its dynamics are charucterized by a function J. which models the variations in the distribution.
Theorem 1. (Continuous Evolution of Model Parameters) Given Assumption [ follows that the purameters Ot of the predictive model g(: 0c) also evolve continuously over time, and its dynamics are jointly determined by the current state of the model and the function f.
Proof. The temporal derivative of the functional space g(:;0;) 02) represents its evolution direct response to the changes in the conditional probability distribution. Without loss of generality, the ground-truth functional space can be modeled by an ordinary differential equation:
(g 0z) [(( 0z), t).
(1)
Applying the chain rule to decompose the temporal derivative of g:
d
d0, (0.) 30,; dl
0g d0
/ (()00)) = Σ
(2)
di

---

where (8b) is the Jacobian matrix of with respect to 8,. By equating the Eq. 2 to the expression involving f: (0())) d0, dt = (()
(3)
Assuming that (O.) is invertible, the ground-truth derivative of 0, with respect to time is given by:
d0: dt ((o())))
(4)
It is known from the Proof that the evolution continuity of 0, follows from a differential equation. Eq. 4 iidentifies a dynamic framework for the predictive model parameters, which provides a strong motivation to develop dynamic systems for them.
The principal challenge arises from the unknown dynamics enclosed in f. which prevents getting 8r by direct mathematical computation. Recognizing this, we propose learning based approach model the parameter dynamics from the observed domains. We construct a learnable parameter dynamics function h0,,, 1; o). parameterized by d, optimized to topological conjugation between the model dynamics and the data dynamics. Topological conjugation guarantees that the parameter's orbits faithfully characterize the underlying data dynamics, thereby ensuring the model's generalization capability across all time. Specifically, as illustrated in Fig. 2 of the Data Flow and the Model Flow, assuming a conceptual perfect mapping & from the data space to the parameter space, topological conjugation suggests that the composition of the learned dynamic h after should behave identically to applying& & after the dynamics f. i... h of = oo f holds, where o represents function composition. In the CTDG problem, the objective of conjugation is to optimize the model dynamics h and the predictive parameters 0 simultaneously, to minimize the discrepancy between the dynamically derived parameters from h and those obtained through direct training, formulated as follows:
arg min Σ E0
10,00
(5)
where || ||2 denotes the Euclidean norm, and each 0, is determined by:
0: arg min L(Y,, (X,;;())
(6)
and 0f-" is defined as the integral parameters at I, obtained from Lj < 4:
05-0 0, + [" ((0,,T:0)
Here, symbolizes the loss function tailored to the prediction task. By employing Eq. 7 the model dynamics are defined for all observation time, and e be obtained by optimizing Eq. 5 and Eq. 6 via gradient descent. After that, the parameters of the predictive model can be calculated at any specific time using ODE solvers.
4.2 Modeling nonlinear model dynamics by Koopman operators
The aforementioned learning approaches are, however, limited in efficient modeling and prediction due to the entangled nonlinear dynamics in the high-dimensional parameters space. Identifying strongly nonlinear dynamics through such space is particularly susceptible overfitting (8), leading to future predictive models eventually diverging from bifurcation to chaos. Despite the complexity of h, it is governed by the data dynamics f, which are typically low-dimensional and exhibit simple, predictable patterns. This suggests that the governing equations for h are sparse within the over- parameterized space, and thus allow for more simplified and manageable representation in properly transformed space. Motivated by this, we propose leveraging the well- well-established Koopman Theory
24. to represent these complex parameter dynamics. Koopman Theory provides a method for the global linearization of any nonlinear dynamics. It expresses the complex dynamic system as an infiitt-dimensiionll Koopman operator acting on the Hilbert space of the system state measurement functions, in which the nonlinear dynamics will become lincarized.
5

---

To facilitate this approach. the target is to identify set of intrinsic coordinates ip that spans Koopman-invaricn subspace. As illustrated in Fig. 2a of the Model Flow and Koopman Flow, this transformation maps the parameters 0 into latent space = = (0)), with resides in a low n-dimensional space, TR"", where the dynamics become more linearized. We also aim to find a fiite-dimensional approximation of the Koopman operator, denoted as K. E IR*x,, that acts on the latent space and advances the observation of the state to the future : 2 + Z. After that we have:
dp Kip(0).
8
The z dynamics are easier track because they principleize the dynamical system while maintaining its characteristics. It gives us a tighter representation of the parameters within a linear space, which allows us to learn the simple K operator instead of the complex coupled dynamics h(: o). Following the transformation, the dynamics of the parameters can be expressed by:
+ 2j + I"d K2t, dr,
where 2j = (03)).
Finally, an inverse transformation provided by 0:- 03- - the original parameter space. The relational among 0, K., (p, 4 and the Koopman invariant subspace are bounded by a series of loss functions detailed in the next Section.
that maps Koopman space back to
4.3 Joint optimization of model and its dynamics with prior knowledge
We introduce a comprehensive optimization approach designed to ensure that the system accurately captures the dynamics of the data. This process requires the joint optimization of several intercon- nected components under the optional constraint of additional prior knowledge about the underlying dynamics: the predictive model parameters 01.r, the transformation functions yp and و- and the Koopman operator K. Our primary objectives are threefold: 1) Ensuring high prediction accuracy, 2) Maintaining consistency of parameters across different representations and transformations, and 3) Learning the Koopman invariant subspaces effectively. Fig. 2]b) illustrates the role of each constraint within our system: we manage two sets of states, intrinsic and integral, aligning across three spaces.
Predictive Model Parameters (f): Each observation time corresponds
predictive model, which tasked with making predictions and serving as the initial values for the dynamical system. They primarily optimized to minimize the prediction error Lintri of different domains:
Lintri = >'_
T L(Vr,
(10)
Koopman Network Parameters (ye, pp_1, IC): We estimate function that transforms the parameters of the predictive model into Koopman space. Meanwhile, these predictive model parameters must be stable and consistent when converted between representations. This realized by three constraints:
1. Reconstruction Loss: An autoencoder is leveraged to map parameter space Koopman space by encoder and back to the original space by decoder -p Liecon ensures consistency between and its reconstructed form via transformations:
Liecon -'' (
(11)
2. Dynamic Fidelity Loss: This term ensures that the transformation produces a Koopman invariant subspaces in which the dynamics can forward correctly. It measures the fit of the Koopman operator K. against the transferred parameter:
Layna E'T Σ j=l || of
112
(12)
where 2, = (00,) and أا = + J.y Kiz, dr.
3. Consistency Loss: It measures the consistency between the original and the dynamic parameter
in the model space:
Leonsis ΣT T Σ j=1 110,, 12, where of =
(13)

---

Additionally, we load the dynamically integral parameters 6 +i back into the predictive model to evaluate its predictive capability, quantified by Linteg =
Ej= C(Y,, g(x;; 0 1)). Finally,
the system optimizes the following combined loss to refine all components simultaneously:
{01:T, Pp
argmin (aLintrii + aLinteg + |Lrccon + YLdyne + Lconsie) (14)
0I.T-P.P 1,C
with a, 3, and y as adjustable weights to balance the magnitude of each term, ensuring that no single term dominates during training.
During inference, given the moment ts, the model uses the state from the nearest observation moment Labs as an initial value, integrating over the time interval [Lota; (x] in Koopman space give the generalized model state 8ohs->s at the desired test moment.
Analysis and Control of the Temporal Domain Generalization. Integrating Koopman theory into continuous temporal domain modeling facilitates the application of optimization, estimation, and control techniques, particularly through the spectral properties of the Koopman operator. We remark that the Koopman operator serves as a pivotal interface for analyzing and controlling the generalization process. Constraints imposed on the Koopman space will be equivalently mapped to the Model space. For instance, the eigenvalues of K. are crucial as they provide insights into the system's 's stability and dynamics, illustrated below:
2j + I"" Kz, dr, where X is an eigenvalue of KC
(15)
1. System Assessment. The generalization process is considered stable if all A satisfy Re(A,) < 0. Conversely, the presence of any A such that Re(A:) > 0 indicates instability in the system. When Re(A,) = 0, the system may exhibit oscillatory behavior. By analyzing the locations of these poles in the complex plane, we can assess the system's 's long-term dynamics. helping us identify whether the generalized model is likely to collapse in the future.
2. Behavioral Constraints. Adding explicit constraints to IC can guide the generalization toward desired behaviors. This process not only facilitates the incorporation of prior knowledge about domains but also tailors the system to specific characteristics. To name just a few, if the data dynamics are known to be periodic, configuring K. as K B-B" BT,, with B as learnable parameters, ensures that the model system exhibits consistent periodic behavior since the eigenvalues of
B - BT are purely imaginary values. Additionally, employing low-rank approximation such as K. = UVT, with U, V E R and l < n, allows the model to concentrate on the most significant dynamical features and explicitly control the degrees of freedom of the generalization process.
Theoretical Analysis. In this work, we theoretically proved that our proposed continuous time TDG method has smaller or equal error compared to the discrete-time time method for approximating temporal distribution drift. This demonstrates that the ODE method provides a more accurate approximation due to its consideration of the integral of changes over time, reducing the accumulation of errors compared to the step-wise updates of the discrete-time methods.
Theorem 2 (Superiority of continuous-time methods over discrete-time methods (informal)). Continuous-time methods have smaller or equal errors compared to discrete-time methods in ap- proximating temporal distribution drift, due its consideration of the integral of changes over time.
The formal version and proof of Theorem are given in Appendix We also provide a detailed model complexity analysis in Appendix A.1.4.
5 Experiment
In this section, we present the performance of the Koodos in comparison to other approaches through both quantitative and qualitative analyses. Our evaluation aims at I) assessing the generalizability of Koodos in continuous temporal domains; 2) assessing whether Koodos captures the correct underlying data dynamics; and 3) assessing whether Koodos can use inductive bias to guide the behavior of the generalization. Detailed experiment settings (i.e., dataset details, baseline details, hyperparameter settings, ablation study, scalability analysis, and sensitivity analysis) are demonstrated in Appendix A.T. Besides, since the TDG a special case of the CTDG, we also conducted experiments on the traditional discrete temporal domain generalization task. Results are shown in Appendix B
7

---

Table 1: Performance comparison on continuous temporal domain datasets. The classification tasks report Error rates (%) except for the AUC for the Twitter dataset. The regression tasks report MAE, 'N/'' implies that the method does not support the task.
Classification
Regression
Model
2-Moons
Rot-MNIST
Twitter
Yearbook
Cyclone
House
Offline
13.5 + 0.3
6.6 + 0.2
0,54 + 0.09
8.6 + 1.0
18.7 + 1.4
19.9 + 0.1
LastDomain
55.7 + 0.5
74.2 0.9
0,54 0.12
11.3 1.3:13
22.3 1 0.7
20.6 0.7
IncFinetune
51.9 0.7
57.1 +: 1.4
0.52 :.000.001
11.0
19.9
20.6 1 0.2
IRM
15.6 0.2
8.6
0.11
0.5
08
19.8 + 0.2
V-REx
12.8 0.2
8
0.58
0.
17.7
20.2 + 0.1
CIDA
18.7 4 2.0
0.7
0.03
8.4
17.0 0.4
10.2 + 1.0
TKNets
39.6
37.7
0.04
0.3
N/A
N/A
DRAIN
53.2 0.9
19.1
0.04
1.0
23.6
23.6::55 0.5
9.8 = 0.1
DRAIN-A
46.2 0.8
57.24 1.8
0.59 0.02
26.2 1.2
4.6
9.9 = 0.1
DeepODE
17.8 5.6
48.6
0.64 0.02
1.1
18.5
+ 3.3
10.7 = 0.4
Koodos (Ours)
2.8 + 0.7
4.6 0.1
0.71 0.02
6.6 1.3
16.4 4 0.3
9.0 + 0.2
Datasets. We compare with classification datasets: Rotated Moons (2-Moons), Rotated MNIST (Rot-MNIST), Twitter Influenza Risk (Twitter), and Yearbook; and the regression datasets: Tropical Cyclone Intensity (Cyclone), House Prices (House). More details can be found in Appendix [A.I.. A..II
Baselines. We employ three categories of baselines: Practical baselines, including 1) Offline; 2) LastiDomain; 3) IneFinetune; 4) IRM M: 5) V-REx (25). Discrete temporal domain generalization
methods, including 1) CIDA 448;; 2) TKNets [57]: 3) DRAIN 22; 4) DRAIN-At. Continuous
temporal domain generalization methods, including 1) DeepODE [1]. Comparison method details
can be found in Appendix A.112
Metrics. Error rate (%) is used for classification tasks. As the Twitter dataset has imbalanced labels. the Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC) curve is used. Mean Absolute Error (MAE) is used for regression tasks. All models were trained on training domains and then deployed on all unseen test domains. Each method's ss experiments were repeated five times, with mean results and standard deviations reported. Detailed parameter settings for each dataset are provided in Appendix A.1.3.
5.1 Quantitative analysis: generalization across continuous temporal domains
We first present the performance of our proposed method against baseline methods, highlighting results from [ Koodos exhibits outstanding generalizability across continuous temporal do- mains. A key observation is that all baseline models struggle to handle synthetic datasets, particularly challenged by the continuous and substantial concept drift (i.e., 18 degrees of rotation per second). In real-world datasets, methods like CIDA, ARAIN, and DeepODE demonstrate effectiveness in certain cases. However, the performance gap between them and Koodos highlights the importance of explicitly considering continuous temporal modeling. For instance, while DRAIN attempts to address domain dynamics through probabilistic sequential approach via LSTM, this introduces considerable errors due to the inherent discrete temporal. Moreover, while DRAIN-At and DeepODE adjust for temporal irregularities accordingly, they fail to adequately synchronize the data and model dynamics, leading to unsatisfactory results. In contrast, Koodos establishes promising approach and a benchmark in CTDG tasks, with quantitative analysis firmly confirming the approach's 's superiority.
5.2 Qualitative analysis: data dynamics and the learned model dynamics
We conducted a qualitative comparison of different models by visualizing their decision boundaries on the 2-Moons dataset, depicted in Fig. 3 Each row represents a different method: DRAIN-At. DeepODE, and Koodos, with the timeline at the bottom tracking progress through test domains. DRAIN-At displays the highest error rate, showing substantial deviation from the anticipated trajectories, especially after the third domain. We also observe that DRAIN-At seems to freeze when predicting multiple steps, likely due to its underlying model, DRAIN, uses recursive training strategy within a single domain and is explicitly designed for one-step prediction. DeepODE shows a relatively better performance. benefits from leveraging ODEs to maintain continuity the model dynamics. However, the nonlinear variation of the predictive model parameters complicates its ability to abstract and simplify the real dynamics. Its predictions start close to the desired paths but diverge over time. Finally, Koodos exhibits the highest performance with clear and concise boundaries,
8

---

DRAIN-GC Bir: 44.6%
DesPODE 10.3%
Roodos Brr: 2.1%
Domain 43.0138
Test
Domain
Domain 47,4
76.7882
8994
44,2791
4796
Last Train Dumain
150722
Test Timeline
Figure 3: Visualization decision boundary of the 2-Moons dataset (purple and yellow show data regions, red line shows the decision boundary). Top to bottom compares two baseline methods with ours; left to right shows partial test domains (all test domains are marked with red points on the timeline). All models are learned using data before the last train domain.
Tratice Stddd Exabaddin
Medel Ieeepekiie Medel Extoptatii
Last Train Domsin
Tine
50
10
Finst
-100
-50
-50
-30_20 10
100
-100
10
30
(a) Koodos
DRAIN
(c) DeepODE
Koodos captures the essence of generalization through the harmonious synchronization of model and data dynamics; Middle: DRAIN, as a probabilistic model, fails to capture continuous dynamics, which is presented as jumps from one random state to another. Right: DeepODE demonstrates certain degree of continuity, but the dynamics are incorrect.
Figure 4: Interpolated and extrapolated predictive model trajectories. Left:
consistently aligning with the actual dynamics and maintaining high fidelity across all tested domains, showcasing its robustness in continuous temporal domain modeling and generalization.
Fig. 4 (a) demonstrates the space-time evolution of the generalization process of Koodos. By applying I-SNE, the predictive model parameters are reduced to 2-dimensional representation space, plotted against the time on the Z-axis. We used Koodos to interpolate the model states (35 seconds displayed in blue-yellow line) among the training domain states (marked by blue docs) in steps of 0.2 seconds, and similarly extrapolated 75 steps (15 seconds displayed in red line). The visualization clearly shows that Koodos synchronizes the model dynamics with the data dynamics: the interpolation creates cohesive, aoward-spiraling trajectory transitioning from the first the last training domain, while the extrapolation correctly extends this trajectory outward into a new area, demonstrating the effective of Koodos from another intuitive way. We also show the space-time evolution of baseline models in Fig. bbc)), in which we do not find meaningful patterns of the generalization process.
5.3 Analysis and control of the generalization process
The learned Koopman operator provides valuable insights into the dynamics of generalized predictive models. We analyze the behavior of the learned Koodos model on the 2-Moons dataset, focusing on the cigenvalues of the Koopman operator. As illustrated in Fig. 5la), the eigenvalues are distributed

---

Stable Bagian (Be-c))
Complex Figenvelues
Unstable Region (Res-O)
Purcly Seal
Purely Imaginary
Complex Plane
Complex Piane
2
Imaginary
0.2
Imaginary
0.0
0.0-0000-
0-
-0.2
-0.4
-0.6
-100
Real (A)
Real (1)
Without Inductive Bias
(b) With Inductive Bias
Figure 5: Eigenvalue distribution of the Koopman operator. Left: K. as learnable; Right: K. = B-BB- Br
with B learnable.
Trosiing Model Exbeeliigg
tnaeppattionn
Modil Extrpsiation
Long Extrapolation
35
Time
120
Time
Last Train Domain
70
00
Long Extrpollton
-1 10
First Train Domaig
10
Last Train Domain
35
First Train Domain
10
20
-10
-0
20
Without Inductive Bias
(b) With Inductive Bias
Figure 6: Extremely long-term extrapolated predictive model trajectories in uncontrolled and con- trolled settings. Left: IC learnable; Right: K. = B 3² with learnable.
across both stable (Re<O) and unstable (Re>0) regions on the complex plane. The spectral distribution suggests that while Koodos performs effectively across all tested domains, will eventually diverge towards instability and finally collapse. To validate, extended the extrapolation of the Koodos an extremely long-term (i.e., 35 seconds future). Results depicted in Fig. ((a) demonstrate that the generalized model's trajectory significantly deviates from the anticipated spiral path, suggesting that extremely long-term generalization will end with the accumulation of errors.
Fortunately, Koodos's innovative design allows it to incorporate knowledge that extends beyond the observable data. By configuring the Koopman operator K. B BT, ensure that all eigenvalues of the final learned K. are purely imaginary (termed Koodos ). promoting stable and periodic behavior. This adjustment is reflected in Fig. ((b), where the eigenvalues tightly clustered around the imaginary axis. As shown in Fig. bb), the embeddings and trajectories of Koodos cohesive and maintain stability over extended periods. Remarkably, even with 85 seconds of extrapolation, Koodos+ shows no signs of performance degradation, highlighting the significance of human inductive bias in improving the robustness and reliability of the generalization process.
6 Conclusion
We tackle the problem of continuous temporal domain generalization by proposing continuous dynamic system network, Koodos. We characterize the dynamics of the data and determine its impacts on model dynamics. The Koopman operator is further learned represent the underlying dynamics. We also design a comprehensive optimization framework equipped with analysis and control tools. Extensive experiments show the efficiency of our design.
Acknowledgments and Disclosure of Funding
This work was supported by Japan Science and Technology Agency (IST) SICORP, Grant Number JPMISC2104 104; JST SPRING, Grant Number JPMISP2108: the SPRING GX Overseas Dispatch Program and the Shibasaki Scholarship Foundation.
10

---

References
[1] Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk mini- mization, arXiv preprint 1907.0290.09 2019.
[2] Guangji Bai, Chen Ling, and Liang Zhao. Temporal domain generalization with drift-aware dy- namic neural networks. In The Eleventh International Conference on Learning Representations, 2023.
[3] Guangji Bai, Johnny Torres, Junxiang Wang, Liang Zhao, Cristina Abad, and Carmen Vaca. Sign-regularized multi-task learning. In Proceedings of the 2023 SIAM International Conference on Data Mining (SDM), pages 793-801. SIAM, 2023.
[4] Guangji Bai, Qilong Zhao, Xiaoyang Jiang, Yifei Zhang, and Liang Zhao. Saliency-guide hidden associative replay for continual learning. arXiv preprint arXiv:2310.04334 2023.
[5] Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain generalization using meta-regularization. Advances in neural information processing systems, 31, 2018.
[6] Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 3722-3731, 2017.
[7] Steven L Brunton, Marko Budišié, Eurika Kaiser, and J Nathan Kutz. Modern koopman theory for dynamical systems. arXiv preprint (rXi::2102.12080 2021.
[8] Steven L. Brunton, Joshua L Proctor, and Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 133(15):3332-337, 2016.
[9] Zekun Cai, Renhe Jiang, Xinyu Yang. Zhaonan Wang, Diansheng Guo, Hill Hiroki Kobayashi. Xuan Song, and Ryosuke Shibasaki. Memda: Forecasting urban time series with memory-based drift adaptation. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 193 202, 2023.
[10] Boyo Chen, Buo-Fu Chen, and Hsuan-Tiee Lin. Dotation-blender cnns on a new open dataset for tropical eyclone
regression. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 90-99, 2018.
[11] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
[12] Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 2916):14-1142 2012.
[13] Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain gener- alization via model-agnostic learning of semantic features. Advances in neural information processing systems, 32, 2019.
[14] Yingjun Du, Jun Xu, Huan Xiong, Qiang Qiu, Xiantong Zhen, Cees GM Snoek, and Ling Shao. Learning to learn with variational information bottleneck for domain generalization. In European Conference on Computer Vision, pages 200-216, 2020.
[15] Yuntao Du, Jindong Wang, Wenjie Feng, Sinno Pan, Tao Qin, Renjun Xu, and Chongjun Wang. Adarnn: Adaptive learning and forecasting for time series. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management (CIKM), 2021.
[16] João Gama, Indré Žliobaité, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. A survey concept drift adaptation. ACM computing surveys (CSUR), 46(4):1-37, 2014.
[17] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversariil training of neural networks. Journal of machine learning research, 17(59):1-35, 2016.

---

[18] Rui Gong, Wen Li, Yuhua Chen, and Luc Van Gool. Dlow: Domain flow for adaptation and generalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2477-2486, 2019.
[19] David Francis Griffiths and Desmond Higham. Numerical methods for ordinary differential equations: initial value problems, volume 5. Springer, 2010.
[20] Huan Hc, Owen Queen, Teddy Koker, Consuelo Cuevas, Theodoros Tsiligkaridis, and Marinka Zitnik. Domain adaptation for time series under feature and label shifts. In International Conference on Machine Learning, pages 12746-12774, PMLR, 2023.
[21] Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross- domain generalization. In 16th European Conference Computer Vision, ECCV 2020, pages 124-140. Springer, 2020.
[22] Junteng Jia and Austin R Benson. Neural jump stochastic differential equations. Advances in Neural Information Processing Systems, 32, 2019.
[23] Patrick Kidger, James Morrill, James Foster, and Terry Lyons Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33:6696-6707, 2020.
[24] Bernard Koopman. Hamiltonian systems and transformation in hilbert space. Proceedings of the National Academy of Sciences, 17(5)3115311, 1931.
[25] David Krueger, Ethan Caballero, Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk ex- trapolation (rex). In International conference machine learning, pages 5815-5826. PMLR, 2021.
Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai
[26] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta- learning for domain generalization. In Proceedings of the AAAI conference on artificial intelligence, volume 38, 2018.
[27] Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy Hospedales. Sequential learning for domain generalization. In European Conference on Computer Vision, pages 603-619. Springer, 2020.
[28] Da Li. Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pages 5542-5550, 2017.
[29] Jingjing Li, Zhiqi Yu, Zhekai Du, Lei Zhu, and Heng Tao Shen. A comprehensive survey on source-free domain adaptation. IEEE Transuctions on Pattern Analysis and Machine Intelligence, 2024.
[30] Wen Li, Zheng Xu, Dong Xu, Dengxin Dai, and Luc Van Gool. Domain generalization and adaptation using low rank exemplar svms. IEEE transactions on pattern analysis and machine intelligence, 4015):111411127, 2017.
[31] Wendi Li, Xiao Yang, Weiqing Liu, Yingce Xia, and Jiang Bian. Ddg-da: Data distribution generation for predictable concept drift adaptation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 4092-4100, 2022.
[32] Alexander H Liu, Yen-Cheng Liu, Yu-Ying Ych, and Yu-Chiang Frank Wang. A unified feature disentangler for multi-domain image translation and manipulation. Advances in neural information processing systems, 31, 2018.
[33] Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Joao Gama, and Guangquan Zhang. Learning under concept drift: A review. IEEE transactions on knowledge and data engineering, 31(12):2346 2363, 2018.
[34] Massimiliano Mancini, Samuel Rota Bulo, Barbara Caputo, and Elisa Ricci. Best sources forward: domain generalization through source-specific nets. In 2018 25th IEEE international conference image processing (ICIP), pages 1353-1357. IEEE, 2018.
12

---

[35] Massimiliano Mancini, Samuel Rota Bulo, Barbara Caputo, and Elisa Ricci. Adagraph: Uni- fying predictive and continuous domain adaptation through graphs. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6568 6577, 2019.
[36] Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Dis- secting neural odes. Advances in Neural Information Processing Systems, 33:3952-3963, 2020.
[37] Saeid Motiian, Marco Piccirilli, Donald Adjeroh, and Gianfranco Doretto. Unified deep supervised domain adaptation and generalization. In Proceedings of the IEEE international conference on computer vision, pages 5715-5725, 2017.
[38] Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International conference on machine learning, pages 10-18. PMLR, 2013.
[39] Anshul Nasery, Soumyadeep Thakur, Vihari Piratla, Abir De, and Sunita Sarawagi. Training for the future: A simple gradient interpolation loss to generalize along time, Advances in Neural Information Processing Systems, 34:19198-19209, 2021.
[40] Fengchun Qiao, Long Zhao, and Xi Peng. Learning to learn single domain generalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12556-12563 2020.
[41] Tiexin Qin, Shiqi Wang, and Haoliang Li. Generalizing to evolving domains with latent structure-aware sequential autoencoder. In International Conference on Machine Learning, pages 18062-18082. PMLR, 2022.
[42] Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for irreuulaty-samppldd time series. Advances in neural information processing systems, 32, 2019.
[43] Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada. Asymmetric tri-training for unsupervised domain adaptation. In International conference on machine learning, pages 2988-2997. PMLR, 2017.
[44] Mona Schirmer, Mazin Eltayeb, Stefan Lessmann, and Maja Rudolph. Modeling irregular time series with continuous recurrent units. In International Conference on Machine Learning, pages 19388-19400 19405. PMLR, 2022.
[45] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Picter Abbeel. Domain randomization for transferring deep neural networks from simulation the real world. In 2017 JEEE/RS. international conference on intelligent robots and systems (IROS), pages 23 30. IEEE, 2017.
[46] Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, Varun Jampani, Cem Anil, Thang To, Eric Cameracci, Shaad Boochoon, and Stan Birchfield. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. Proceedings of the IEEE conference computer vision and pattern recognition workshops, pages 969-977, 2018
[47] Erie Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167-7176. 2017.
[48] Hao Wang, Hao He, and Dina Katabi. Continuously indexed domain adaptation. In International Conference Machine Learning, pages 9898-9907. PMLR, 2020.
[49] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Yu Philip. Generalizing to unseen domains: A survey on domain general- ization. IEEE transactions on knowledge and data engineering, 355558805-- 2022.
[50] Mei Wang and Weihong Deng. Deep visual domain adaptation: survey. Neurocomputing, 312: 13:1135-55,, 2018.
13

---

[51] G Wilson and DJ Cook. A survey of unsupervised deep domain adaptation. ACM Transactions on Intelligent Systems and Technology, 11(5):1-46, 2020.
[52] Mixue Xie, Shuang Li, Longhui Yuan, Chi Liu, and Zehui Dai. Evolving standardization for continual domain generalization over temporal drift. Advances in Neural Information Processing Systems, 36, 2024.
[53] Zihao Xu, Guang- Yuan Hao, Hao He, and Hao Wang. Domain-indexing variational bayes: Interpretable domain index for domain adaptation. In Eleventh International Conference Learning Representations (ICLR), 2023.
[54] Huaxiu Yao, Caroline Choi, Bochuan Cao, Yoonho Lee, Pang Wei W Koh, and Chelsea Finn. Wild-time: benchmark of in-the-wild distribution shift over time. Advances in Neural Information Processing Systems, 10110099-103222 2022.
[55] LIN Yong, Fan Zhou, Lu Tan, Lintao Ma, Jianmeng Liu, HE Yansu, Yuan Yuan, Yu Liu, James Y Zhang. Yujiu Yang, al. Continuous invariance learning. In The Twelfth International Conference Learning Representations, 2023.
[56] Dazhou Yu, Guangji Bai, Yun Li, and Liang Zhao. Deep spatial domain generalization. In 2022 IEEE International Conference on Data Mining (ICDM), pages 1293-1298. IEEE, 2022.
[57] Qiuhao Zeng, Wei Wang, Fan Zhou, Gezheng Xu, Ruizhi Pu, Changjian Shui, Christian Gagné, Shichun Yang, Charles X Ling, and Boyu Wang. Generalizing across temporal domains with koopman operators. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 16651-16659, 2024.
[58] Liang Zhao, Qian Sun, Jieping Ye, Feng Chen, Chang Theng-Tien Lu, and Naren Ramakrishnan. Feature constrained multi-task learning models for spatiotemporal event forecasting. IEEE Trunsactions on Knowledge and Data Engineering, 29(5): 2017.
14

---

A Appendix
A.1 Experimental details
A.1.1 Dataset details
In this paper, we explore a variety of datasets analyze the performance of machine learning models under conditions of continuous temporal domain. We employ the following datasets:
1. Rotated 2-Moons This dataset is variant of the classic 2-entangled moons dataset, where the lower and upper moon-shaped clusters are labeled 0 and 1, and each contains 500 instances. We randomly generated 50 distinct timestamps from the continuous real number time interval [0, 50). Each timestamp corresponds to a specific domain, with each domain created by rotating the moons 18 counterclockwise per unit of time. The continuous concept drift represented by the progressive positional changes of the moon shon-shaped clusters.
2. Rotated MNIST This dataset is a variant of the classic MNIST dataset [2], comprising images of handwritten digits from 0 to 9. We randomly generated 50 distinct timestamps from the continuous real number time interval [0, 50). Each timestamp corresponds to a specific domain, within which we randomly selected 1,000 images from the MNIST dataset and rotated them 18 counterclockwise per unit of time. Similar to 2-Moons, the continuous concept drift means the rotation of the images.
3. Twitter The Twitter dataset [58 3 utilizes tweet data to predict flu intensity. We randomly collected streaming tweets starting at 50 arbitrary timestamps lasting 7 days during the 2010-2014 flu seasons and then examined the volume of disease-related terms to predict current flu trends. The result is validated against the Influenza- levels reported by the Centers for Disease Control and Prevention (CDC). The continuous concept drift in this dataset is characterized by the fluctuations in tweet volumes over the years and the variation in flu activity throughout the season.
Illness (ILI) activity
4. Yearbook The Yearbook dataset [54] consists of frontal-facing yearbook portraits collected between 1930 and 2013 from 128 high schools. The task is to classify gender from images. We randomly sampled 40 years of data from the 84-year dataset, with each year representing a domain to represent the incomplete temporal domain collection process, which resulted in variable time intervals between consecutive domains. The Yearbook datasel provides visual record of evolving fashion styles and social norms across the decades.
5. Cyclone The Cyclone dataset [T0] is collected by satellite remote sensing and is dedicated to the task of tropical cyclone imagery to wind intensity. When each cyclone occurs, the satellite collects a series of images for its entire life cycle as a domain, with the date of its occurrence representing a temporal domain time. We focused on cyclone data from the West Pacific region covering 2014 to 2016 and formed 72 continuous domains. The dataset is event-triggered, and the variation in wind strength associated with the seasonal dates results in continuous concept drift.
6. House This dataset comprises housing price data from 2013 2019 and is utilized for regression task to predict the price of house given the features. We extracted sales data for one-month durations across 40 arbitrary time periods from 2013 through 2019. The concept drift in this dataset is how the housing price changes over time for a certain region.
To more clearly illustrate the temporal arbitrariness of the continuous temporal domain, we plot the time distribution of domains for each dataset in Fig. 7 where the blue line represents the timestamp of one specific domain. We use the last 30% domain of each dataset as test domains, which are marked with gray shading.
A.1.2 Comparison methods
1. Practical Baseline (1) Offfine Model: This model operates without temporal variations and is trained using Empirical Risk Minimization (ERM) across all source domains. (2) LastDomain: This model also trained without temporal variations but focuses solely on
the last source domain using ERM. (3) IncFinetune: This approach begins by applying the baseline method at the initial time point and subsequently fine-tunes the model each following time point using a lower learning rate. (4) IRM: IRM is to train predictive model
15

---

Domain Moments the 2-Moons Dataset
10
25
20
25
30
35
40
45
50
Time
Domain Moments of the Rot-MNIST Dataset Domain Moments of the YearBook Dataset
10
20
25
35
50
Time
Domain Moments of the Influenza Dataset
175
16
24
32
40
48
30
64
72
80
Time
Domain Moments of the Cyclone Dataset
40
120
200
280
Domain Moments the House Dataset
250
500
750
3000
1230
1500
1750
2000
2250
Figure 7: The distribution of continuous temporal domain datasets along the time dimension. The test domains are marked with gray shading.
over all distributions that do not rely on spurious correlations much as possible. (5) V-REx: Like IRM, V-REx performs causal identification, while also providing some robustness to changes in the covariate shift by risk extrapolation.
2. Discrete Temporal Domain Generalization Baseline Continuous temporal domain tasks require consideration of domains for the long future rather than a limited number of steps. Existing methods are mainly unable to deal with this problem. Among them, although DRAIN (2) is designed only for predicting one-step future prediction, its internal LSTM structure has the potential of long-term future prediction, so we implemented DRAIN as a discrete baseline. We implemented two versions of it: (1) DRAIN: This approach tackles the temporal domain generalization problem by proposing a dynamic neural network, which uses an LSTM to capture the evolving pattern of neural network, during the test, DRAIN will consider all future test domains as one domain, and utilizes the LSTM to predict the future parameters of this target domain. (2) DRAIN-AS At: A variant of DRAIN, in which the internal LSTM is changed to a continues recurrent units [44] to model irregular distributed domains. Under this setting, DRAIN-AN At treats the test domains and predicts the model state at that given moment. We also added CIDA [48] and TKNets
irregular time series
[57]] comparisons. (3) CIDA treats the domain index continuous variable to help the discriminator using a distance- based loss. (4) TKNets trains a prediction model for the target domain by leveraging the evolving pattern among domains.
3. Continuous Temporal Domain Generalization Baseline There is currently no method to generalize the model over the continuous time domain. However, we constructed a method that can infer continuous states using a neural dynamical system. (I) DeepODE: We train deep neural Ordinary Differential Equations (NeuralODEs) [ to model the continuous dynamics of the predictive model parameters. DeepODE consists of three structures. first uses an encoder to embed the model parameters as dense vectors, then uses NeuralODEs to infer their state in the desired future, and later uses decoder to transfer the state back to the parameters. DeepODE treats the first domain as an initial value and infers the state of future training domains at once, using the inferred model's performance on training domains to optimize its autoencoder and NeuralODEs. In the testing phase, when a specific moment is given, we use DeepODE to infer the prediction model parameters at that moment, and then reload the estimated parameters into the prediction model for prediction.
A.1.3 Model configuration
We use autoencoder to achieve the y and -1, which we call Encoder and Decoder in the following. linear transformation layer without bias (named Dynamic Model) is served as the Koopman operator. In certain complex data dynamics, the Koopman operator needs to expand its dimensions (potentially up to infinity) for better approximation of the dynamics. Since it's not feasible to increase the
16

---

dimensions endlessly, we approximate the infinitely wide Koopman operator using multiple linear layers with ReLU. In practice, we found that two linear layers work well enough. Additionally, Predictive Models for each domain are trained to provide initial parameter values for Generalized Models. We use NeuralODEs [T] as the ODEs solvers. For large models with massive parameters, we treat the head network as feature extractor shared by all domains, inferring only the state of the remaining layers. All experiments are conducted on a 64-bit machine with two 20-core Intel Xeon Silver 4210R CPU @ 2.40GHz, 378GB memory, and four NVIDIA GeForce RTX 3090. We use Adam Optimizer for all experiments, and we specify the architecture as well as other details for each dataset follows:
1. Rotated 2-Moons The Predictive Model consists of 3 hidden layers, with dimension of 50 each. We use ReLU layer after each layer and Sigmoid layer after the output layer. The Encoder and Decoder are both a 4-layer Multi-Layer Perceptron (MLP), with dimensions of [1024, 512, 128, 32] for each layer. A 32-dimensional linear layer Dynamic Model serves as the Koopman operator. The learning rate for the Predictive Model is set at 1 x 10-?, while for the others is set X 10-3.
2. Rotated MNIST The Prediction Model features convolutional neural network architecture comprising three convolutional layers with channel [32, 32, 64] and a kernel size of 3. Each convolutional layer is followed by ReLU layer and a max pooling layer with kernel size of 2. Following the flattening, the network includes two linear layers with dimensions [128, 10). A dropout layer is added between the linear layers to prevent overfitting. We treat the convolutional layer as the feature extractor. The Encoder and Decoder are both a 4-layer MLP with dimensions of [1024, 512, 128, 32). A 32-dimensional two linear layers Dynamic Model serves as the Koopman operator. The learning rate for all parts is set at 1 x 10-3.
3. Twitter The Prediction Model consists of 3 hidden layers, with dimension of [128, 32, 1]. We use the ReLU layer after each layer and a Sigmoid layer after the output layer. The Encoder and Decoder are both a 4-layer MLP, with dimensions of [1024, 512, 128, 32] for each layer. A 32-dimensional two linear layers Dynamic Model serves as the Koopman operator. The learning rate for all parts is set at 1 x 10-3
4. Yearbook The Prediction Model features a convolutional neural network architecture com- prising three convolutional layers with channel (3,,32, 64) and kernel size of 3. Each convolutional layer is followed by a ReLU layer and max pooling layer with a kernel size of
2. Following the flattening, the network includes three hidden linear layers with dimensions [128, 32, 1]. A dropout layer is added between the linear layers to prevent overfitting. The convolutional layer is used the feature extractor. The Encoder and Decoder are both 4-layer MLP with dimensions of [1024, 512, 128, 32]. A 32-dimensional two linear layers Dynamic Model serves as the Koopman operator. The learning rate set at 10-3
3
5. Cyclone The Prediction Model features convolutional neural network architecture com- prising four convolutional layers with channel [32, 32, 64, 64] and a kernel size of 3. Each convolutional layer is followed by a ReLU layer and max pooling layer with kernel size of 2. Following the flattening, the network includes three hidden linear layers with dimensions [128, 32, 1]. We treat the convolutional layer as the feature extractor. The Encoder and Decoder are both a 4-layer MLP, with dimensions of [1024, 512, 128, 32] for each layer. A 32-dimensional 2 linear layers Dynamic Model serves as the Koopman operator. The learning rate for all parts is set at 1 x 10-3
6. House The Prediction Model consists of 3 hidden layers with a dimension of 400. We use the ReLU layer after each layer. The Encoder and Decoder are both a 4-layer MLP with dimensions of [1024, 512, 128, 32). A 32-dimensional two lincar layers Dynamic Model serves as the Koopman operator. The learning rate for all parts is set at x 10 1
A.1.4 Complexity analysis
In our framework, coordinate transformations between spaces are implemented using an autoencoder achieved by MLP. The complexity of these transformations is described by a linear relationship in terms of N, expressed as ((2(Nd + E) + F), where N denotes the number of parameters e in the predictive models, denotes the number of neurons in the first MLP layer, E denotes for the number of parameters remaining in the autoencoder, and F denotes the number of parameters in the Koopman operator. In many deep learning tasks, significant portions of the model's layers function as representation learning and can be shared, resulting in small N and controlled model complexity.
17

---

Table 2: Ablation test results for different datasets.
Ablation
2-Moons
Rot-MNIST
Cyclone
X Linteg
27.4 + 20.3
42.7 # 1.0
55.0 + 1.0
X Lyecon
3.2
0.3
17.2 + 0,4
X dgnna
5.1
23.3 +.3:4. 28.9
X L consis
30.5 18.4
5.1
17.4 0.5
XK.S
7
OOM
0OM
Koodos
2.8 + 0.7
4.6 = 0.1
16.4 + 0.3
A.1.5
Ablation study
conducted an ablation study to systematically assess the contribution various constraints and components within Koodos two classification datasets: 2-Moons, Rot-MNIST, and regression dataset Cyclone. The results are summarized in the Table 2 For each test, specific loss constraints removed, indicated by X, and also tested the effect of bypassing the Koopman Space but learning dynamics directly in the parameter space, designated as XK.S.
As shown in the table, each component of constraints contributes to robust performance across all tested datasets. The removal of particular elements has marked effects on model stability and accuracy. Specifically, the absence of Linteg and Ldyna constraints leads to significant performance degradation, highlighting their essential roles in learning the correct dynamics of the model through continuous temporal domain data. Removing other constraints leads to unpredictable results on certain datasets decline in effectiveness, indicating their importance in ensuring system effectiveness and stability under various input conditions. The considerable increase in error observed with the XK.S on the 2-Moons dataset demonstrates the challenges associated with modeling nonlinear dynamics directly in the parameter space, and the out-of-memory (OOM) problems with Rot-MNIST and Cyclone upon the removal of the Koopman Space suggest that this component is not only pivotal but also computationally demanding.
Computation Time vs. Parameter Size
Computation Time vs. Number of Domains
Reletive Time
Relative Time
1.3
1.2
1.1
1.0
1c3es
5e5
206
3c0
100
250
500
750
1000
Predictive Model Parameter Size
Number of Domains
(a) Scalability for Parameter Size
(b) Scalability for Domain Number
Figure 8: Scalability analysis w.r.t the number of parameters and the number of domains.
Table 3: Cost of training and testing time.
Model
Train Time (s)
Test Time (s)
DRAIN
67
<|
DecpODE
289
CIDA
610
3,05
TKNets
948
Koodos
566
2.83
A.1.6 Scalability analysis
We conducted comprehensive analysis to evaluate the scalability of our system concerning both the number of parameters the predictive model and the number of domains, as shown in Fig. Computational times are normalized relative to the shortest time to provide consistent basis.
18

---

To explore how the size of predictive model parameters affects runtime, we experimented with various configurations by varying the depth and number of neurons in the model. These configurations were tested on the Cyclone dataset with parameter counts ranging from 2,000 to over one million. As depicted in Fig. X a), there is relatively linear and low growth rate in computation time as the parameter size increases. This gradual increase suggests that the autoencoder in Koodos to encode the parameter space into a Koopman space effectively mitigates the impact of increased parameter scale, maintaining computational efficiency even predictive model complexity grows.
To assess the effect of domain count on runtime, we generated synthetic 2-Moons datasets with varying domain numbers: 10, 50, 100, 250, 500, 750, and 1000 domains, each with 200 training instances categorized into two labels. The runtime is plotted against the number of domains in Fig. B), demonstrating a linear increase computational time. This linear progression aligns with expectations for system processing a sequence of input domains through ODEs, indicating predictable and manageable scalability as domain count increases.
We further conduct the model scalability analysis by comparing the running time of Koodos with other state-of-the-art baselines: DRAIN, DeepODE, CIDA, and TKNets in the 2-Moons dataset. As shown in Table our model strikes a good balance between training time and effectiveness.
2-Moons
2.Wocrs
2-Moons
Error Rates (S)
10.
73
30
2.5
2.5
7.5-
33
300
500
10
40
10
138 356 112 10000000
House
2.0
0.2
Bad
MAR
94
3.4
9.2
9.2
2.
100
522 10240005
Figure 9: Sensitivity analysis.
A.1.7 Sensitivity analysis
We conducted the sensitivity analysis on 2-Moons and House datasets to understand the hyperparam- eters in Koodos; loss weights (a, 3, 7) and the dimensions n of the Koopman operator AC. The loss weights are set based on the magnitude of each loss term. For instance, the 2-Moons dataset, the cross-entropy loss Lintri and Lintey are approximately 1 after convergence, the Tyecon and Lconsiis in the Model space are around 0.01, and Ldyna in the Koopman space around 0.1. Accordingly, we set the initial values of a, B, y to 1, 100, and 10, respectively. We then adjust each term independently within its respective range: a and y are varied from to 100, and |3 from 10 to 1000. The dimension of the Koopman operator varies from 16 to 2048. Fig. 9 shows the results.
It can be seen that: (1) Koodos exhibits stable behavior with controlled variance across a wide range of hyperparameter values, evidencing its insensitivity to hyperparameter variations. (2) Setting the loss weights of (a, 3,) by the magnitude of each loss term is sufficient for achieving good performance, and fine-tuning the weights may give better results. (3) A hundreds-dimension approximation of the Koopman operator is sufficient to achieve good results. Too high may lead to overfitting.
A.1.8 Limitations
The limitation arises from the assumptions inherent in the research area of the temporal domain generalization framework. TDG assumes that domain pattern drifts follow certain predictable, smooth patterns, allowing for modeling future changes as a sequence. CTDG extends TDG to continuous time, further generalizing the problem to handle domains collected at arbitrary times. Thus, their assumptions are aligned in focusing on smooth, predictable concept drift. To address other kinds of drift, alternative frameworks may be required. However, different frameworks can be combined to address the full spectrum of domain generalization challenges. Exploring such a comprehensive approach may represent promising research direction.
19

---

Table 4: Performance comparison on discrete temporal domain datasets. The classification tasks report Error rates (%), and the regression tasks report MAE.
Classification
Regression
Model
2-Moons
Rot-MNIST
ONP
Shuttle
House
Appliance
Offiin
22.4 + 4.6
18.6 = 4.0
33.8 + 0.6
0.77 0.1
11.0 = 0.36
10.2 = 1.1
LastDomain
14.9 0.9
17.2
36.0 : 0.2
0
10.3 0.16
9.1 0.7
IncFinetune
16.7 + 3.4
10.1
34.0 0.3
0.83 + 0.07
9.7 + 0.01
8,9 + 0.5
CDOT
9.3 + 1.0
+
34.1 0.0
0.94
CIDA
10.8 1.6
9.3+0.7
3..7
9.7 + 0.06
8.7 0.2
GI
3.5 1.4
=
0.84
0.29
9.6
8.2
DRAIN
3,2 1.2
115111
3.2
0.26
9.3 0.14
6,4 0.4
Koodos (Ours)
1 0.4
7.0
33.5 0.4
0.24
8.8 0.19
4.8 0.3
B
Experiments on discrete temporal domain generalization
While our primary focus is on Continuous Temporal Domain Generalization (CTDG), we have also conducted experiments in the more conventional context of Temporal Domain Generalization (TDG). In essence, TDG can be seen as a specialized case of CTDG, where making the continuous system with step size fixed at 1. Such setting does not affect the proper ability of the CTDG model to capture the data dynamics and the corresponding model dynamics.
In TDG experiments, we follow the problem definition in DRAIN data domains is collected over discrete time intervals. Each domain, denoted as Dr corresponds to dataset collected
2 Formally, sequence of
specific time where t 1,2, .... We have sequence of domains {D1, D.,..., Dr}. where each domain D: {(*), of m consists of N; instances with features of X; and labels yf E Y,, and X, and Yú represent the random variables for features and targets. The goal in TDG the dynamics of (01. .,,...,ry) then predict the 0r+1 for the unseen future domain Dr+1 at time t 7'+ finally do the prediction task - = ((T+1i 0T+1).
train a predictive model g (X; 0.) on these historical domains, and modeling
We have replicated the experimental setups typically used in TDG research. Specifically, we used datasets, settings, and hyperparameters from the DRAIN 22] We compare with the following classification datasets: Rotated Moons (2-Moons), Rotated MNIST (Rot-MNIST), Online News Popularity (ONP), and Shuttle; and the following regression datasets: House prices dataset (House), Appliances energy prediction dataset (Appliance). Note that the first two datasets differ from datasets used in our CTDG main experiments in Exp. 5 as here they are time-regularly distributed and only have 1-step future domain to predict. We keep the architecture of the predictive model for each dataset the same DRAIN [2].
Performance comparison of all methods in terms of misclassification Error (in %) for classification tasks and mean absolute error (MAF) for regression tasks. All experiments are repeated times for each method, and we report the average results and the standard deviation in the quantitative analysis.
The results are shown in Table The Koodos model consistently outperforms all baselines across various datasets, demonstrating its robustness and superior adaptability also made traditional temporal domain generalization tasks. Notably, Koodos significantly reduces error margins, especially real-world datasets such as House and Appliance. The exceptional performance of Koodos can be attributed to its fundamental continuous dynamic system design, which effectively captures and synchronizes dynamic changes data and models.
20

---

C Theorem proofs
Assumptions
Continuous Data Distribution Drift: Let pt)) be the data distribution at time t. Assume p(t) is continuously differentiable, and the drift de(t) is Lipschitz continuous with constant L.
2. Training and Target Domains: The model is trained on a sequence of training domains {p(t))}=z where t t2 ... tr. The target domain at a future time s, with the data distribution p(s). The time intervals At, f ft can be arbitrary.
3. Model Training: For the ODE model, let ar(1) be the state at time I, governed by the ODE:
dx(t)
(x)
dt
where f is a smooth function. For the RNN model, let h4, be the state at discrete time t,, updated by:
he.. = hr, + At, (hh,, T..),
where e is smooth function and Tt, is the input at time 4,
Generalization Error: The generalization error on the target domain p(s) depends on the accumulated training errors and the error propagation from the last training domain p(tr) to the target domain p(s).
Theorem 3 (Formal version of Theorem 2). Given the assumptions, continuous method using an Ordinary Differential Equation (ODE) provides smaller or equal generalization error unseen target domain compared to a discrete-tine method using a Recurrent Neural Network (RNN), due to its more accurate approximation of the data distribution drift over time.
Proof. Below is the formal proof:
Accumulated Training Error
1. ODE Model: The state xt + At;) can be approximated by the ODE integral:
+
x (t, + At,) = =(t;) +
f(x(T),r)dr.
(16)
J:
Using the Taylor series expansion for (xx(1), ():
(At)) 0f
r(t: + At,) x(t,) + f(((tt), (.) At)
Ot + +(((t)))))
(17)
The error for one step is:
ErrorODE, step (((At)))).
(18)
TTe accumulated error over T steps is:
ErroroDE, train
Σ (((A))))
(19)
==1
Given that the intervals At, can be large, the accumulated error becomes:
T-1
ErroroDE, tiin = Σ (AL))
(20)
2. RNN Model: The state heas is updated as:
hett he, + At, ((he,,
(21)
Assuming he, se x(tt) and o(h, Ix,)) xe f(x(t.), t.). ), the error for one step is:
ErToIRNN, step O(A4).
(22)
The accumulated error over T steps is:
Σ O(At))
ETTOTRNN,
(23)
Giien that the intervals At, can be large, the accumulated error becomes:
EITOTRNN, tmin =0
Σ A
(24)
21

---

Error Propagation to Target Domain
ODE Model: To handle an arbitrary future time 8, we integrate the error over many small steps from by to 8:
ErToTODE, train + [ O((T (((T-t))?) dr.
ErrOFODE, target
(25)
Since ranges from ty to s:
ErrotoDE, target
ErroroDE, train O((s )T)
(26)
RNN Model: Similarly, the error propagation the target domain involves many small steps:
K-1
ErToTRNN, target R EITOTRNN, train + O(Ax)):
(27)
-where K is the number of small steps from by and At, are the small intervals:
EIroTRNN, target ErrOTRNN, unin O(s tr).
(28)
Generalization Error Comparison
For the ODE model:
ErroFoDE target O Σ (At;)) O(((s )))))
(29)
For the RNN model:
ERroIRNN, target O
Σ Sti Ed:
+0(s - tr).
(30)
SSnce (A4)2 is much smaller than A for any Ari and (s 4T) is much smaller than 8 lr, we have:
ErroroDE, target & ErTorRNN, target-
(31)
22

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims made in the abstract and introduction accurately reflect our paper's s contributions and scope.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have included a section to discuss our paper's limitation in Appendix
A.L.8
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model -peccficciii,nonn. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tes system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer: [Yes]
Justification:
We provided all assumptions and proofs for theories in our paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provided comprchensive information well the source code of our method.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provided open-access data and code to reproduce our experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //iiss.cc/ppll wwtpcccpppppppppeeeeeeeeeeeeeeee httpa//gggeeee/eeeeubbiisssssss /Cc////oddiiisssseeeeeee ssiiiiiisiyyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provided the necessary details to run our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We are closely following existing literature in reporting the experimental results and error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provided the necessary details on the computing resources we used in our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpp:://neurpps.cc http://s....cpp..ttth//cccccleee
Answer: [Yes]
Justification:
Our paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no potential negative societal impact found in this work. Our research is foundational research and not tied to particular applications.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly credited and cited licenses of existing assets.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
27

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provided proper documents for released code and data.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moddl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve these.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "8Um75D82PN", "reviewer": "Reviewer_2Dt9", "summary": "This paper introduces a new task: Continuous Temporal Domain Generalization (CTDG) to address the limitations of traditional TDG in handling continuously evolving and irregularly observed temporal data. By proposing the Koopman operator-driven framework (Koodos), this work leverages Koopman theory and optimization strategies to learn and control the continuous dynamics of data and models. Experiments demonstrate the effectiveness and efficiency of Koodos in managing complex high-dimensional nonlinear dynamics and generalizing across continuous temporal domains.", "questions": "1. Does CTDG still adhere to the assumption of smooth distribution shifts?\n\n2. Is the quantitative evaluation in the paper still performed on the last domain?\n\n3. Regarding the Yearbook dataset under CTDG evaluation settings, is it essentially still discrete but can be approximated as continuous due to its large number of domains? If so, which other datasets in CTDG use dense discrete domains to approximate true continuous data?", "limitations": "See my weakness part.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces a valuable new task, Continuous Temporal Domain Generalization (CTDG), which moves beyond the discrete nature of traditional TDG tasks, enhancing their relevance to real-world applications.\n\n2. The paper proposes the Koopman operator-driven framework (Koodos), effectively leveraging Koopman theory and optimization strategies to understand and control the continuous dynamics of both data and models.\n\n3. It expands the evaluation setting of traditional TDG tasks by introducing datasets with truly continuous data and temporal distribution shifts. This setting encompasses the previous discrete TDG evaluation as a special case, providing a more comprehensive evaluation framework for all TDG methodologies.\n\n4. The authors further evaluate their method and other TDG baselines under this new CTDG evaluation setting, achieving significantly improved performance.\n\n5. The paper is well-written and well-organized.", "weaknesses": "1. Evaluation is limited to small datasets and models, lacking assessment on higher-dimensional datasets, which limits the practical applicability of proposed benchmarks and obscures performance in scenarios with large models and high-dimensional data.\n\n2. The continuous nature of CTDG proposed here appears similar to task settings in CIDA and AdaGraph, which focus on domain adaptation. This reduces the novelty of the task setting in this paper, necessitating further discussion and experimental comparisons for clarification.\n\n3. TKNets, another Koopman theory-based TDG method, is relevant but lacks clear differentiation through discussion and experimental comparisons.\n\n4. Including information on the training and inference costs of each method would enhance the completeness of the study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for reviewing and acknowledging our work! We see your main concerns are about related works and evaluations. Please read our answers along with the Rebuttal PDF.\n\n> W1. Limited evaluation on small datasets/models; lacks high-dimensional datasets.\n1. We provide a summary of the datasets and predictive models used in our work in Tab. 5. From it, it is evident that we have utilized high-dimensional datasets and modest-sized datasets and models in our evaluation: the Yearbook dataset comprises around 20k images with 32*32=1024 pixels, and the model used for this dataset has approximately 163k parameters. Similarly, the Cyclone dataset, with image dimensions of 64x64x2=8192, is tested with a model having 135k parameters. \n2. As a comparison, the existing discrete TDG datasets benchmark only uses data below 100 dimensions, and the minimum size of the model parameter is 12k.\n3. Combined with the intrinsic challenge in TDG field, further increasing model size to a very large scale (e.g., LLMs) is a promising open area that is considered as a great future direction.\n\n> W2. CTDG's continuous nature resembles CIDA and AdaGraph settings, needing further discussion and comparisons.\n1. Our work differs from their setup in the following points:\n    - **Different Fields**: CIDA and AdaGraph focus on domain adaptation with access to target domain data during training. CTDG extends TDG and focuses on domain generalization without accessing target domain.\n    - **Different Semantics**: the term 'continuous' is used differently in each work. CIDA: Domain index as a continuous variable instead of a categorical variable to aid the discriminator with a distance-based loss. AdaGraph: Continuous arrival of test data and online adaptation. CTDG: the temporally sequential domains are collected on **continuous-time** instead of fixed interval discrete-time.\n    - **Different Objectives**: CIDA and AdaGraph follow traditional DA/DG settings without modeling temporal evolution dependence like CTDG. Although modeling such temporal patterns has been theoretically shown to have a lower generalization bound [50].\n    - CTDG's novelty lies in its unique requirenment to capturing temporal dependencies and generalizing in continuous-time temporal domains, a challenge not addressed by CIDA, AdaGraph, or existing TDG methods.\n2. Modifications to CIDA's then can apply to our dataset. We implemented it.  Tab. 4 shows that CIDA cannot deal well with the CTDG problem.\n\n> W3. TKNets need further discussion and comparisons.\n1. Our work and TKNets have different starting points, frameworks, and contributions.\n    - **Focus and Motivation**: Our work addresses domain evolution over continuous-time, which existing TDG methods, including TKNets, fail to capture effectively as they rely on discrete-time models (e.g., state-space model, LSTM).\n    - **Central Contribution**: Our primary contributions are theoretical proof that the model for continuous temporal domains is also continuous, based on which we construct a dynamic system described by differential equations. The Koopman operator is used to simplify learning complex differential equations. In contrast, TKNets' core contribution is proposing to use the Koopman operator to model the domain state transition matrix, serving a different role from ours.\n2. We implemented TKNets as a baseline for the CTDG task. Results can be checked in Tab. 4 and it shows TKNets cannot deal well with the CTDG problem.\n\n> W4. Including the training and inference costs would be better. \n\nWe have added related experiments; please check Tab 6. Our model strikes a good balance between training time and effectiveness.\n\n> Q1. Does CTDG still adhere to the assumption of smooth distribution shifts?\n1. Yes, CTDG extends TDG and retains the smooth assumption.\n2. Moreover, CTDG mitigates two key assumptions of traditional TDG:\n    - train domains are collected at fixed time intervals without missing, e.g., t=1,t=2,t=3\n    - test domains appear only at a time interval in the immediate future, e.g., t=4 \n3. Unlike these assumptions, CTDG allows training domains to be collected at arbitrary times, e.g., t=1.2, t=2.432, t=5.4693…, and adapts to test domains that appear at any point in the future, e.g., t=6.324, t=8.3, t=9.45, not limited to immediate subsequent steps, nor limited to the future one domain.\n\n\n> Q2. Is the quantitative evaluation in the paper still performed on the last domain?\n\n1. No. As we answered in Q1, one important task of CTDG is to generalize the model to any future point. Therefore, we perform evaluations in multiple test domains arbitrarily and irregularly distributed after the training period.\n2. For the number of test domains for each dataset, one can refer to the Tab. 5, while the time distribution of these multiple test domains can be found in Figure 6, marked with gray shading.\n\n> Q3. Is the Yearbook dataset discrete but dense as continuous? How about others?\n1. No. We do not use dense discrete domains to approximate continuous data. All datasets are designed to be consistent with real-world CTDG case situations: (1) Discrete-Time Domains but with Missing Domains; (2) Event-Driven Domains; (3) Streaming Data with Arbitrary Data Collecting Times.\n2. All three of these scenarios require CTDG. Therefore, the datasets are designed as follows:\n    - The YearBook dataset represents (1). The raw data is collected by years; we randomly sampled 40 years from 83 years to represent the incomplete temporal domain collection process.\n    - The Cyclone dataset represents (2). When each cyclone occurs, the satellite collects a series of images for its entire life cycle, with the date of its occurrence representing a temporal domain time.\n    - Situation 3 is simulated by setting multiple randomly started data collection windows on the tweet stream and the price stream, to create Influenza dataset and House dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "olYTVef2lQ", "reviewer": "Reviewer_xaRP", "summary": "The paper presents a novel approach called Continuous Temporal Domain Generalization (CTDG), addressing the challenge of training predictive models under continuously evolving and irregularly observed temporal domains. Unlike traditional TDG methods that rely on discrete time intervals, CTDG captures continuous dynamics of both data and models. The proposed Koopman operator-driven continuous temporal domain generalization (Koodos) framework leverages Koopman theory to learn underlying dynamics and enhances it with a comprehensive optimization strategy. Extensive experiments demonstrate the effectiveness and efficiency of the proposed approach.", "questions": "The article repeatedly emphasizes the importance of \"C\" in CTDG and the arbitrary selection of time t. However, does the proposed method specifically address this aspect, or does it primarily rely on Koopman theory?\n\nIn the methodology section, the article mentions that domain conditional probability distributions will not experience abrupt changes. \nHowever, this situation holds true only when the sampling time intervals are equal. The method addresses arbitrary sampling; does this imply it also deals with abrupt changes?\n\nNotably, in equation (15), the weights of L_pred and L_pred2 are both α, the weights of L_recon and L_consis are both β, and the weight of L_dyna is γ. Could you explain the rationale behind this? How are these hyperparameters set for different datasets, and what is the basis for their selection? Do different weights significantly affect the results?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a novel problem (CTDG) and proposes an innovative solution (Koodos) leveraging Koopman theory.\nThe methodology is sound and well-executed. And it is well-supported by robust theoretical foundations and demonstrates excellent performance in the experimental results.", "weaknesses": "The explanation of how the article addresses domain changes with arbitrary temporal sampling is not very clear.\n\nThere is some overuse of formula characters, and the use of L_pred2 seems rather arbitrary.\n\nAdditionally, the numerous hyperparameters in the loss function increase the cost of tuning for different datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for reviewing and acknowledging our work! Please read along with the Rebuttal PDF.\n\n>W1. The explanation of how the article addresses domain changes with arbitrary temporal sampling is not very clear.\n1. TDG assumes there is widespread smooth, predictable distribution changes over time, so the field aims to predict future distributions and models by building dynamic models with time-varying parameters, aligning the model function with predicted future domain distributions. CTDG extends TDG, relaxing the requirement for fixed-interval discrete-time domain collection and test not limited to immediate one-step future domains.\n2. Inferring future model parameters under CTDG is challenging. In our work, we prove that if the domain distribution changes continuously, then the corresponding predictive model parameters should also change continuously. This insight allows us to formulate a differential equation for the predictive model parameters. Through the differential equation, we can predict future model parameters at any continuous time.\n3. However, analytically solving this difference equation is almost impossible. Note that the Koopman operator allows us to linearize the nonlinear dynamics of the difference equation by mapping the original nonlinear predictive model parameter space into a higher-dimensional linear space. By leveraging this operator, we can effectively solve the differential equations numerically, ensuring that the model parameters evolve precisely over time and generalize well to the far future.\n4. Finally, the learned differential equation will directly compute the predictive model parameters in any specific future tume.\n\n>W2. There is some overuse of formula characters, and the use of L_pred2 seems rather arbitrary.\n\n1. Before submitting for review, we carefully checked the use of symbols and tried our best to make them concise and clear. We apologize for any confusion. We will carefully review the manuscript again and try to streamline the notation to ensure greater clarity and readability.\n2. The loss function L_pred2 is used to quantify the prediction error of dynamically integral parameters performed on domain data. L_pred2 works together with L_pred to constitute the evaluating term of the integral and intrinsic model parameters on the domain task. We acknowledge that the notation might be unclear. To improve understanding, we will rename L_pred and L_pred2 to L_intrinsic and L_integral, respectively. This change will more accurately reflect their roles.\n\n> W3 & Q3. Additionally, the numerous hyperparameters in the loss function increase the cost of tuning for different datasets.\n\nThe loss weights are just designed to balance the magnitude of each loss term, ensuring that no single term dominates the model's training process. \n\n1. We conducted a sensitivity analysis to understand the hyperparameters in Koodos: loss weights $(\\alpha, \\beta, \\gamma)$ and the dimensions $n$ of the Koopman operator $\\mathcal{K}$. Fig. 10 (in rebuttal PDF) shows the results.\n2. The loss weights are set based on the magnitude of each loss term. For instance, in the 2-Moons dataset, the cross-entropy loss $L_{pred}$ and $L_{pred2}$ are approximately 1 after convergence, the $L_{recon}$ and $L_{consis}$ in the Model Space are around 0.01, and $L_{dyna}$ in the Koopman Space is around 0.1. Accordingly, we set the initial values of $\\alpha, \\beta, \\gamma$ to 1, 100, and 10, respectively. We then adjust each weight term independently within its respective range: $\\alpha$ and $\\gamma$ are varied within 1 to 100, and $\\beta$ within 10 to 1000. The dimensions $n$ of the Koopman operator vary within the range of 16 to 2048.\n3. It can be seen that: \n- Koodos exhibits stable behavior and robust convergence across wide varied hyperparameters, evidencing its ease of training, robust convergence, and insensitivity to hyperparameter.\n- Setting the loss weights in accordance with the magnitude of each loss term is sufficient for achieving good performance.\n- A few hundred Koopman operator dimensions are sufficient good; too many can lead to overfitting.\n\n>Q1. The article repeatedly emphasizes the importance of \"C\" in CTDG and the arbitrary selection of time t. However, does the proposed method specifically address this aspect?\n\n1. As we discussed in W1., CTDG extends TDG, relaxing the requirement for fixed-interval discrete-time domain collection and test not limited to immediate one-step future domains. Defining tasks in continuous time is challenging because it breaks the discrete-time models (e.g., state-space models, LSTM) that traditional TDG relies on. \n2. Therefore, we theoretical proof that the parameter of time-varing model for continuous temporal domains is also continuous, leading to a dynamic predictive model described by differential equations.\n3. Solving differential equations analytically is almost impossible. The Koopman operator simplifies learning these complex differential equations, enabling effective numerical solutions.\n4. By the learned differential equations, the parameters of the predictive model can be calculated at any specific time using ODEssolvers.\n\n>Q2. In the methodology section, the article mentions that domain conditional probability distributions will not experience abrupt changes. However, this situation holds true only when the sampling time intervals are equal. The method addresses arbitrary sampling; does this imply it also deals with abrupt changes?\n\nThe assumption that domain conditional probability is continues changes is based on the nature of the underlying processes being continuous. The observation intervals, whether equal or arbitrary, do not influence the continuity of the underlying processes themselves. For example, the temperature change throughout a day is a continuous process, regardless of whether measurements are taken every hour or at irregular intervals. The continuity of the temperature change remains inherent to the process itself.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h0LwrMRo5J", "reviewer": "Reviewer_Drn2", "summary": "The article introduces Continuous Temporal Domain Generalization (CTDG), which extends traditional Temporal Domain Generalization (TDG) methods by addressing the challenges posed by continuous and irregularly spaced temporal domains. The authors propose a Koopman operator-driven framework (Koodos) to handle data evolving continuously over time and collected at arbitrary intervals. The framework aims to capture continuous dynamics in data and models, optimize generalization across continuous temporal domains, and leverage prior knowledge of dynamic patterns. The experiment results demonstrate the efficacy of this approach compared to some existing methods.", "questions": "1. In typical domain generalization challenges, the objective is to align the joint distribution \\(P(X, Y)\\) across different domains. For Temporal Domain Generalization (TDG), this would extend to aligning \\(P(X(t), Y(t))\\) at different time points. How does Koopman Theory address this issue in aligning these temporally varying distributions? \n\n2. Authors focuses on the existence of continuous and relatively stable dynamic changes across continuous time, how effective is this method when dealing with continuous but non-periodic or irregular data variations or if a new dataset significantly differs in its statistical properties from the training data? is there potential to integrate other mechanisms or models to enhance the handling of such substantial data changes? \n\n3. In the Methods section, you emphasize addressing changes in the conditional probability distribution over time. However, the datasets used, such as Rotated MNIST, Rotated 2-Moons, Yearbook, and Cyclone, are commonly associated with covariate shifts rather than shifts in the conditional probability distribution. These datasets appear to maintain a stable relationship between the labels and the input data or a latent feature space. Could you clarify how the changes in the conditional probability distributions are represented or accounted for in these datasets?", "limitations": "There is no potential negative societal impact is found in this work.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The primary contribution is the introduction of the CTDG problem and the Koodos framework, which extends TDG to handle continuous and irregular temporal domains. The paper also provides a comprehensive optimization strategy and theoretical analysis to support the proposed approach.", "weaknesses": "The method assumes data dynamics can be well-characterized by the Koopman operator. While powerful for linearizing nonlinear dynamical systems, its effectiveness across various domains (e.g., finance, healthcare, social media) requires more extensive empirical studies, especially for highly noisy or chaotic systems. The proposed hybrid loss function's complexity may present challenges in training and convergence. A detailed analysis of its behavior and optimization strategies would be beneficial. The method introduces additional complexity and hyperparameters. A comprehensive discussion on hyperparameter sensitivity and guidelines for setting loss weights would enhance the method's practical applicability.\n\nTheorem 2, which compares continuous-time and discrete-time approaches in approximating temporal distribution drift, does not directly address generalization bounds or risks in domain generalization. While it provides insights into error accumulation over time, it lacks analysis of the model's performance on unseen temporal domains.\n\nThe experimental comparisons are somewhat limited. Only one continuous baseline is used, and it appears that key comparison metrics are missing. Methods like AdaRNN[1] and Diversify[2], which aim to address the same issue, are not discussed or compared. More comparisons to state-of-the-art discrete TDG methods adapted to the continuous case would strengthen the evaluation.\n\n[1] AdaRNN: Adaptive Learning and Forecasting of Time Series \n[2] Diversify to Generalize: Learning Generalized Representations For Time Series Classification", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for reviewing. \n\n> W1. needs more empirical studies.\n> \n1. The Koopman operator is proven theoretically that can linearize any nonlinear dynamical system in [4].\n2. Our empirical evidence supports its applicability across various domains. Datasets are summarized in Tab. 5\n    - Experiments span a wide range of real-world domains (7 in total): social media, culture, climate, economics, business, physics, and energy.\n    - Experiments includes various modalities (tabular, image, text) and tasks (classification, regression).\n    - Experiments use high-dimensional and complex datasets. : Yearbook (32x32=1024 pixels), Cyclone (64x64x2=8192 pixels).\n    - Datasets from complex, noisy real-world sources: Influenza (millions of tweets for flu prediction), Cyclone (satellite images for weather systems).\n3. Koodos demonstrates robustness and significant success across all 12 datasets, which evidence supports the effectiveness of it and highlights its applicability across various domains.\n\n\n> W2. loss function\n\nWe conducted three rounds of convergence tests on the 2-moons dataset. Fig. 11 (rebuttal) shows that:\n1. The loss function and its terms converge effectively, showing Koodos' robustness and ease of training.\n2. Each term in the loss functions is easy to train and converge using off-the-shelf optimizers such as Adam without requiring special optimization tricks.\n3. Ablation tests (Section A 1.5) show performance drops when any term is removed, justifying their inclusion.\n\n> W3. method complexity. A hyperparameter sensitivity.\n\nThe loss weights are designed to balance each term's magnitude, ensuring no term dominates training. for example, 2-moons is set 1,100, and 10 as each loss around 1,0.01,0.1. Koodos is designed to be simple, comprising only a Koopman Network and a differential equation solver. We conducted sensitivity analysis, shown in Fig.10 \n1. Koodos exhibits stable behavior and robust convergence across wide varied hyperparameters, evidencing its ease of training, robust convergence, and insensitivity to hyperparameters.\n2. Setting the loss weights in accordance with the magnitude of each loss term is sufficient for achieving good performance.\n3. A few hundred Koopman operator dimensions are sufficiently good;\n\n> W4. Theorem 2\n\nWe analyze the risks associated along the time line based on the cumulative error on the training set, and it will propagate to the test set. So the cumulative error on the test set will also be smaller.\n\n> W5. comparison limited. AdaRNN Diversify not discussed. Including more TDG\n\n1. AdaRNN and Diversify address different problems compared to CTDG/TDG:\n    - **Different Objective**: The core problem AdaRNN and Diversify solved is creating domain-splitting algorithms to solve the problem of lacking predefined domain divisions in time series classification datasets. In contrast, TDG/CTDG have predefined domains, with the primary challenge is capturing the evolutionary pattern among temporally sequential domains and generalize the pattern to future domains.\n2. We recognize that AdaRNN and Diversify used adversarial learning strategy DANN. Therefore, we implemented CIDA, an advanced adversarial learning baseline compared to DANN. We also implemented IRM and V-REx to together show domain-invariant learning effectiveness. Results in Tab. 4 show none of these methods having fair results.\n3. We implemented the state-of-the-art discrete TDG method TKNet. Results in Tab. 4 indicate it does not handle CTDG. Notably, currently no more continuous TDG methods are available for comparison.\n\n> Q1. How does Koopman Theory align temporally varying distributions?\n\n1. TDG differs from traditional DG. TDG assumes there is widespread smooth, predictable distribution changes over time, so the field aims to predict future distributions and models by building dynamic models with time-varying parameters, **aligning the model function with predicted future domain distributions**. CTDG extends TDG, relaxing the requirement for fixed-interval discrete-time domain collection and test domain not limited to immediate one-step future domains. Defining like this challenging because it breaks the discrete-time models (e.g., state-space models, LSTM) that traditional TDG relies on. Therefore, we theoretical proof that the parameter of time-varing model for continuous temporal domains is also continuous, leading to a dynamic predictive model described by differential equations.\n2. Solving differential equations analytically is impossible. So we use Koopman operator to simplifie learning these complex differential equations, enabling effective numerical solutions. \n\n> Q2. How deal with non-periodic/irregular data variations/significantly change? Can we integrate other mechanisms?\n\n1. TDG and CTDG assume future predictability. If domain changes are irregular or unpredictable, this violates the model's biases. Since different models have different implicit biases, we need to find a suitable frameworks whose prior assumptions match the setting, such as domain-invariant learning, ensemble learning, or importance weight sampling with restart.\n2. Our approach complements other DG methods. For example, a static feature extractor from domain-invariant learning can be used in Koodos, or equipted Koodos to onlining system with detection-and-restart strategies. Exploring comprehensive systems is a promising future direction.\n\n> Q3. Is datasets are with covariate shifts?\n\nThe datasets we used represent changes in P(Y∣X) and are widely recognized in the TDG field:\nRotated 2-Moons: Labels change with rotation, e.g., point (1,0) changes from label 0 to 1 after 180 degrees. Rotated MNIST: Rotating '6' to look like '9' should still be labeled '6'. Yearbook: Changing fashion trends cause the same features (e.g., clothing styles) to have different gender labels over time. Cyclone: Changes in atmospheric pressure or temperature cause similar satellite images to represent different storm intensities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YcU8GT2YpX", "reviewer": "Reviewer_nh98", "summary": "The paper introduces the problem of Continuous Temporal Domain Generalization. It extends the Temporal Domain Generalization problem, that aims at developing models under temporally varying data, to handle data collected at arbitrary and continuous time points. The paper proposes a framework based on Koopman theory for this problem. It models the underlying dynamics and proposes an optimization strategy. The paper includes several classification experiments to demonstrate the effectiveness of the proposed approach.", "questions": "* How does it compare to domain invariant approaches (e.g. IRM, V-REx etc.)? A domain-invariant approaches might be better at handling abrupt changes.", "limitations": "* see limitations points in \"Weaknesses\".", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper introduces a new problem continuous temporal domain generalization where data is continuous and irregularly observed. This is relevant and challenging setting. \n* The paper is supported by proofs and the relevant assumptions under which the model is well-defined are clearly stated. \n* The paper includes extensive experiments on various datasets, both synthetic and real-world, to demonstrate the effectiveness of the proposed method. The proposed model improves over existing methods.", "weaknesses": "* the paper could discuss more the limitations of the proposed approach, in particular Assumption 1 that states the conditional probability distributions follow an ODE. While reasonable to make, this assumption is quite restrictive; I understand that the considered problem is challenging so relevant assumptions should be made. However, the paper would need to discuss the validity of this assumption. Also, another limitation can come from abrupt changes throughout time that might not be well captured by a continuous ODE formulation (mentioned in A.1.7.); this assumption is likely to be violated in real-world settings.\n* the paper does not seem to follow the checklist; there is no justification for each points and if I am not mistaken, some \"Yes\" answer are not correct e.g. a Broader Impact section is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for reviewing and acknowledging our work! We see your main concerns are about assumption and evaluations. Please read our answers along with the Rebuttal PDF.\n\n> W1. discuss more the limitations; Assumption 1 that states the conditional probability distributions follow an ODE, need to discuss the validity. abrupt changes throughout time that might not be well captured by a continuous ODE formulation; this assumption is likely to be violated in real-world settings.\n\n1. Assumption 1 does not state that the conditional probability distributions P(Y(t)∣X(t)) follow an ODE. Rather, it assumes that if continuous temporal domains exhibit gradual concept drift, then the conditional probability distributions P(Y(t)∣X(t)) change continuously. In this context, 'change continuously' means that the distribution changes are smooth and incremental. This assumption of continuity is reasonable because, in the field of concept drift research, gradual (or incremental) concept drift is considered to be characterized by small, incremental changes[R1]. While in real-world gradual concept drift is more complex and may imperfectly fit the continuity assumptions, here we do not lose much generality, as the underlying factors leading to gradual concept drift are primarily continuous physical, biological, social, or economic processes.\n2. Given the continuity of gradual concept drift, it is appropriate to use differential equations as a tool for modeling. Differential equations provide a mathematical framework to capture the essence of continuous processes. They are particularly useful when the system's evolution can be represented by smooth and continuous dynamics. We chose to model the continuity of model parameters using ODEs as an initial attempt because ODEs offer a simple yet clear and flexible toolset for understanding how systems evolve over time. While it is possible to replace ODEs with more advanced differential equations (e.g., Partial Differential Equations (PDEs), Stochastic Differential Equations (SDEs)) to incorporate factors such as noise and uncertainty, this may detract from the key focus on modeling the fundamental continuous processes. Indeed, our experiments on both synthetic and real-world data have shown that even the simplest ODEs are enough to achieve a pretty significant success.\n\nDiscussion of Limitation:\n\n1. The limitation regarding capturing abrupt drift arises from the assumptions inherent in the research area of Temporal Domain Generalization (TDG) framework. TDG assumes that domain pattern drifts follow certain predictable,smooth patterns, allowing for modeling future changes as a sequence. CTDG extends TDG to continuous time, further generalizing the problem to handle domains collected at arbitrary times. Thus, their assumptions are aligned in focusing on smooth, predictable concept drift.\n2. To address abrupt drift, alternative frameworks are required. These include domain-invariant learning in static models, or ensemble learning/importance weight sampling with a detect-and-restart strategy. These methods operate under different assumptions and frameworks compared to TDG and CTDG. It can be challenging to determine which model is superior without considering the specific application scenario, as different models come with different implicit biases. \n3. However, different frameworks can be combined to address the full spectrum of domain generalization challenges. Our approach does not conflict with methods designed to handle abrupt drift, such as ensemble learning or detect-and-restart strategies. This integration could create a more robust and adaptable system. Exploring such a comprehensive approach may represent a promising direction for future work.\n\n> W2. the paper does not seem to follow the checklist; there is no justification for each points and if I am not mistaken, some \"Yes\" answer are not correct e.g. a Broader Impact section is missing.\n\nWe apologize for our all answers are yes. We review and justify each point in the checklist to ensure compliance.\n1. Broader Impact: There is no potential negative societal impact is found in this work. our research is foundational research and not tied to particular applications.\n2. Q11, Yes should be NA.\n3. Q13, Yes should be NA.\n4. Q14, Yes should be NA.\n5. Q15, Yes should be NA.\n\n> Q1. How does it compare to domain invariant approaches (e.g. IRM, V-REx etc.)? A domain-invariant approaches might be better at handling abrupt changes.\n\n1. We implemented IRM and V-REx, we also implemented an advanced adversarial learning-based baseline CIDA [42] for comparison. CIDA is a powerful strategy built upon adversarial learning. It enhances domain-invariant representations by considering the distance between domain indexes.  \n2. Results are shown in Table 4 (in rebuttal PDF), demonstrating that our proposed Koodos performs competitively against these methods. The effect of Koodos is still by a huge margin.\n3. However, as we discussed before, different frameworks can be combined to address the full spectrum of domain generalization challenges. We think exploring a comprehensive system is a promising future direction.\n\n[R1] Learning under Concept Drift: A Review, TKDE 2018", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you to each reviewer for their valuable time.
Please download the pdf file before reviewing the responses.

Best,
Authors

Meta-review: This paper focuses on continuous domain generalization. All reviewers are in favor of acceptance, although it is important for the authors to incorporate the requested improvements to the paper.
Decision: Accept (poster)