Latent Trajectory Learning for Limited Timestamps under Distribution Shift over Time
Authors: QIUHAO Zeng
Changjian Shui
Long-Kai Huang
Peng Liu
Xi Chen
Charles Ling
Boyu Wang
Venue: ICLR 2024 oral (2024)
Keywords: Distribution Shift
Temporal Distribution Shift
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: Solving Distribution Shift over Time modelling with Stochastic Differential Equations

Abstract: Distribution shifts over time are common in real-world machine-learning applications. This scenario is formulated as Evolving Domain Generalization (EDG), where models aim to generalize well to unseen target domains in a time-varying system by learning and leveraging the underlying evolving pattern of the distribution shifts across domains. However, existing methods encounter challenges due to the limited number of timestamps (every domain corresponds to a timestamp) in EDG datasets, leading to difficulties in capturing evolving dynamics and risking overfitting to the sparse timestamps, which hampers their generalization and adaptability to new tasks. To address this limitation, we propose a novel approach SDE-EDG that collects the Infinitely Fined-Grid Evolving Trajectory (IFGET) of the data distribution with continuous-interpolated samples to bridge temporal gaps (intervals between two successive timestamps). Furthermore, by leveraging the inherent capacity of Stochastic Differential Equations (SDEs) to capture continuous trajectories, we propose their use to align SDE-modeled trajectories with IFGET across domains, thus enabling the capture of evolving distribution trends. We evaluate our approach on several benchmark datasets and demonstrate that it can achieve superior performance compared to existing state-of-the-art methods.

Published as a conference paper at ICLR 2024
LATENT TRAJECTORY LEARNING FOR LIMITED TIMES- TAMPS UNDER DISTRIBUTION SHIFT OVER TIME
Qiuhao Zeng'
Changjian Shui
Long-Kai Huang
Peng Liu'
Xi Chen
Charles X. Ling'
Boyu Wang
'University of Western Ontario
Vector Institute
UUniversity Toronto 4Noah's Ark Lab
ABSTRACT
Distribution shifts over time are common in real-world machine-learning appli- cations. This scenario is formulated as Evolving Domain Generalization (EDG), where models aim to generalize well to unseen target domains time-varying system by learning and leveraging the underlying evolving pattern of the distribu- tion shifts across domains. However, existing methods encounter challenges due to the limited number of timestamps (every domain corresponds to timestamp) in EDG datasets, leading to difficulties in capturing evolving dynamics and risk- ing overfitting the sparse timestamps, which hampers their generalization and adaptability to new tasks. To address this limitation, we propose a novel approach SDE-EDG that collects the Infinitely Fined-Grid Evolving Trajectory (IFGET) of the data distribution with continuous-interpoaate samples to bridge temporal gaps (intervals between two successive timestamps). Furthermore, by leveraging the inherent capacity of Stochastic Differential Equations (SDEs) to capture continuous trajectories, we propose their use to align SDE-modeled trajectories with IFGET across domains, thus enabling the capture of evolving distribution trends. We evaluate our approach on several benchmark datasets and demonstrate that can achieve superior performance compared to existing state-of-the-ar methods.
INTRODUCTION
Domain Generalization (DG) is fundamental problem in machine learning. It aims to learn model that can perform well on unseen data based on the knowledge learned from multiple related domains AArovssy et al. 2019 Li et al. 2018b Sagawa et al.) 2019; Sun & Saenko, 2016 Chen (ct al.) 2024). However, traditional DG techniques assume that the distribution in different domains remains stationary the distribution of data may shift over time due to factors such as changes in the environment or the underlying system. For example, Age-related changes occur in all ocular tissues, including age-related structural changes in the optic nerve (Grossniklaus et al., 2013). Age-related ocular disease is the most prevalent condition associated with vision impairment and blindness in older adults worldwide (Flaxman et al. 2017). However, the data collected from individuals aged beyond 80 is lacking due to very small sample size, privacy, and other factors. It's necessary to build prediction model based on the age-related pattern learned from the data collected from younger cases.
time, which is often impractical in many real-world scenarios. In practice,
To adapt to changing environments over time, recent research in the community has focused on the scenario of Evolving Domain Generalization (EDG) (Bai et al.) 2023 Nasery et al.) 2021b Qin ct al. 2022 Zeng et al., 2023 Yao al., (0022), aimed to tackle such problems. Specifically, the goal of EDG is to learn and leverage the evolving patterns captured from source domains to achieve generalization capability on the unseen future target domains in a time-varying environment.
However, one fundamental obstacle in existing EDG studies (Bai al. 2023 Qin et al., 2022 Zeng ct al. 2023) is that they suffer from a limited number of timestamps, resulting in overfitting to the sparse timestamp data. Consequently, they cannot properly capture the underlying evolving pattern and extrapolate into the future. In fact, recent research (Mariet & Kuznetsov, 2019) has revealed that
"Corresponding authors: Boyu Wang, Charles X. Ling.

---

Published as a conference paper at ICLR 2024
the sample complexity of time series forecasting tasks scales O(v I/MM, where M is the number of timestamps of the training time-series data.
In this paper, we tackle this problem by constructing a continuously evolving trajectory. Specifically, we create the Infimitely Fined-Grid Evolving Trajectory (IFGET) in the latent representation space, collect the evolving samples by
with two steps: i) firstly, develop sample-to-sample correspondence trajectory of each individual sample; ii) Next, we generate continuous-interoli leveraging such correspondence, aimed bridge the temporal gaps between timestamp intervals and avoid overfitting to sparse timestamps, It is denoted as IFGET, since it is continuous trajectory and thereby can be subdivided into infinitely fine temporal grids. Nevertheless, dealing with continuous trajectories poses another challenge in EDG. Most existing EDG algorithms are designed for discrete timestamps and are not able to handle continuous timestamps, since they employ transition functions to predict data at the next timestamp based on current observation, inherently representing time discretely (Bai et al. 2023 Qin et al.. 2022 Zeng et al. 2023 2024).
To address this issue, we propose to model the temporal dynamics of latent representations by employing Stochastic Differential Equations (SDEs) (Kong et al., 2020 Li et al. 2020; Xu et al.) 2022, Kidger et al.) (02T) to fit the IFGET, which provides natural approach for characterizing continuous-timg stochastic processes. Specifically, we propose a path alignment regularizer, which aligns the latent trajectories characterized by SDEs with the paths generated by IFGET, by maximizing the likelihood of the SDE trajectories based on the observations of IRGET.
To summarize, our proposed algorithm, termed as SDE-EDG, has the following desirable properties
Capturing Evolving Patterns via Infinitely Fined-grid Evolving Trajectory (IFGET) To overcome the limitations of the small number of timestamps in current EDG data, we propose to learn the evolving dynamics by constructing the IFGET. To construct the evolving trajectory, we identify the sample-to-sample correspondence between successive domains and employ interpolation function to generate a continuous-interpolatee sample. IFGET alleviates overfitting to the limited timestamps and improves the generalization distribution shifts over time.
Modelling Trajectories of Latent Representations with Stochastic Differential Equations (SDEs) Leveraging the inherent capacity of SDE to model continuous temporal trajectory, our model aligns the depicted latent trajectories of SDE-EDG with IFGET during the training process. We show that SDE-EDG is capable of quantifying evolving stochastic processes, and theoretically demonstrate that SDE-EDG results in a lower generalization bound for downstream tasks.
RELATED WORKS
Evolving Domain Adaptation / Generalization Evolving Domain Adaptation (EDA) (Hoffman
et al.) 2014 Kumagai & Iwata 2017 Mancini al.] 2019 Liu et al. 2020al Wang et al.] 2020
Kumar et al. 2020; Wang et al. 2022) is a related field that focuses on scenarios where a single labeled domain is available alongside multiple unlabeled intermediate domains. The objective of
EDA is to achieve generalization on unseen target domains. Recently, EDG has received considerable attention from researchers. Approaches to solving the EDG problem can be broadly categorized into two groups The first group parameterizes the learning model with time-sensitive models (Qin et al. 2022] Nasery et al., 2021a Bai et al., 2023). (Qin et al., (022) proposes to tackle the challenges of covariate shift and concept shift, with the probabilistic model incorporating variational inference. The second group (Zeng et al. 2023 2024) maps the source data into future data leveraging evolving patterns. However, these methods still suffer from limited timestamps, which hinder the capture of temporal trends. In contrast, we propose to construct IFGET, where the temporal gaps are filled with interpolations.
Ordinary Differential Equations (ODE) Stochastic Differential Equations (SDE) In recent years, Neural-ODE: (Chen et al. 2018h Sun et al., 2020) have emerged powerful tool for continuous time representation of neural networks. SDES (Øksendal & Øksendal 2003 Kong et al.. 2020; Liu al., (020b) have incorporated stochastic terms into ODE solvers, injecting the model with slight random noise to improve generalization ability and noise robustness. To learn SDE neural
networks' parameters, (Ryder et al. 2018 Xu et al. 2022
al. 2020) use variational inference
Our code is available at http:///thhbbcoo/orarooriiiiiin w0w.
2

---

Published as a conference paper at ICLR 2024
technology and overcome the overfitting problem. SDE has shown excellent performance in machine learning applications, such as generative adversarial model (GAN) (Kidger et al. 2222]] Park et al. 2021a), score-based diffusion model (Song et al. 2020). In this work, we propose novel approach to modeling Evolving Domain Generalization (HDG) as a dynamical system by utilizing SDEs to
effectively represent the continuously evolving trajectory of the data representations, and apply
maximum likelihood to efficiently fit the latent evolving trajectories utilizing SDEs.
PRELIMINARIES
Ordinary Stochastic Differential Equations Neural ordinary differential equations (Neural ODEs) (Chen et al. 2018a) approximate the evolving dynamics with the ordinary differential equation and are defined as
24 = zu + / f(2,,8)ds,
where the hidden state 2ạ E Rd evolves with certain dynamics characterized by a neural network f: IR" + Rd, 2n is the initial state, and s represents time in integrals. An SDE can be regarded as an ODE injected with noise over time:
2g 20 1 ((2.)))ds + J: 9(2.,
(2)
where is latent state that evolves over time, f IRd x R E' is the drift function capture the evolving dynamics, g : Rd x R + Rdx is the diffusion funtion to reflect the uncertainties, and Bg is dddimensional Brownian motion (Wiener Process). SDE has shown superior performance in modeling the dynamical system (Park et al. 2021a Øksendal & Øksendal 22003. Under the EDG settings, the drift function of SDE describes the trends of the distribution shift over time, and the diffusion function models the samples' individual stochastics in their representation space.
Evolving Domain Generalization Let D(x, y, 1) be the probability distribution that characterizes temporal dynamics of an instance & E A' and its label y E y, in which there exist underlying evolving patterns over t. In Evolving Domain Generalization (EDG), we are given M source domains {Sm 1 where Sm { (Iajmms mi)m } 1 is the data of m-th domain sampled from D(x, f at the timestamp fm E [0. T}. and is the sample size of the m-th domain. Note that most existing
works (Bai al. 2023 Nasery et al. 2021b Qin et al. 2022 Zeng et al. (023) assume a constant interval A7 between two consecutive domains. In contrast, our approach exhibits flexibility in tackling the proposed method, we will learn the evolving dynamics in latent space Z, given the practical advantages (Kirchmeyer et al. 2022). e.g., improved discrimination, dimension reduction, and computation resources savings compared to operations in the original input space X. Specifically, for every instance we encode it with feature extractor & + Z, and we obtain the embedded feature z = ②(x) E Z. We focus on the dynamics of D(z, y, throughout this work.
the EDG problem even with irregular time intervals.
The goal EDG to learn a robust and generalizable model from source domains by capturing and leveraging the evolving pattern learned from source domains so that it perform well on the unseen target domains in the path space (Boué & Dupuis 1998) at L timestamps tM e E (T, T I (T* tM+L
min Rin (ho) = min ((..) M.MMM (h) * y].
(3)
/
where v is the distribution of the stochastic path (Boué & Dupuis 1998) of D along timestamps T to T + T 2M+1:M4L and 4:M+1:M4 are short for and at the timestamps (tar+1..... l..I.. R2 is the risk of a learning model he parameterized by parameters 0.
METHODS
With the prior knowledge of SDEs and EDG, we will now formally present our SDE-EDG approach: correspondence, which aids in generation
To build IFGET (section 4.1), we search sample-to-same of continuous interpolations; Neural SDEs models the trajectories of latent representations (Section
4.2); we construct IFGET and employ it as a regularization mechanism to promote the learning of evolving representations while avoiding the acquisition of invariant representations (Section 4.3
3

---

Published as a conference paper at ICLR 2024
4.1 CONSTRUCT INFINITELY FINED-GRII EVOLVING TRAJECTORY
In EDG, datasets have a considerably small size of do- mains/timestamps (e.g., most 30 domains) (Yao et al.] 2022). In contrast, models are trained on the historical data for time-series forecasting tasks spanning over least hundreds of timestamps
Intrrii Kocn
Xm
predict future states (Addison Howard inversion 2020 Yu et al. 2016).
In light of this obstacle, we generate intermediate domains by ap- plying interpolations between two consecutive domains. To ensure such interpolations reflect the evolving pattern of the underlying trajectory over domains, one should have the complete trajectory each individual sample across domains. For example, in weather forecasting, one must have the historical metcorological data of each individual observation station to characterize the climate change trends. Unfortunately. such trajectories usually do not exist in EDG as there is no sample-to-sampe correspondence across domains (e.g., we may not have images of the same person at different age stages), thereby preventing the model from tracking the complete trajectories and extracting the evolving trends. To address this issue, we propose to identify sample correspondence between timestamps, which is
Figure I:
The graphical model depicts the data generation pro- cess of SDE-EDG, where we hide the label superseript À Yn and the sample index X the in- put sample (for example, images) at lm. Yn is the label at lms and Zm is the latent representation tm. The continuous- contnnouu-imeerpooation
Zm++ is interpolated by the sam- ple pair from IFGET.
critical to better alignment of the data structure across domains (Lu et al. 2023 Chen et al. 2022
Blitzer et al. 2006 Das & Lee 2018).
Specifically, for each class k, we take the closest sample at fm+1 to the datapoint zm. at f.n as its subsequent state time 1m+1 to build comple-to-samppee correspondence:
zk 11m+11
argmin Dist( 2//m= >/im+1),
(4)
where Dist: : x 2 - [0, +20) is a distance metric defined over the embedding space, Sk m++1 be the set of Ng data points sampled from Dm=11 (short for D(z, vl( with class y= k E {1..... K} in a training iteration. The rationale here lies in the decomposition of latent variables into class- dependent and domain-dependen
evolving components (Qin et al.) 2022), resulting in samples from the same class in the m-th and (m++)) domains exhibiting smaller distances due to such sample pair's shared class-dependent similarities, while the evolving difference maintains consistent magnitude. Utilizing
correspondence, we gather discrete samples within IFGET. To render continuous, we generate iigiinuouusy-iteppolateee samples bridging the temporal gaps.
With the sample correspondence, we leverage the interpolation function to generate continuous- interpolated samples, such that the interpolation is generated along the approximated individual trajectory of a data point as shown in Figure
(5) where the interpolation rate À 1) sampled from Beta distribution B(B,, 82). 81 and 32 are the parameters of the Beta distribution, and St consists of instances sampled from k-th class of m-th domain. Here we apply linear interpolation (Yan al. 2020 Zhang et al. 2018) as the interpolation function. The continuous-interpolaeee samples bridge temporal gaps in the discrete evolving trajectory, converting into an infinitely fine-graii lne-grained trajectory due to E (0, 1). Specifically, as À can any value between (0, 1), it enables us approximate time moments between the m-th and (m + 1)-th timestamps. We theoretically show that the sample complexity of EDG reduces with a smaller temporal interval in Theorem D.3 which leads to a reduction in error. We take interpolations as approximations of samples between time intervals, leading to a smaller time interval and thus a smaller sample complexity. Above all, we construct the Infinitely Fined-grid Evolving Trajectory { ak zt M + 11m++1)) m=1 by leveraging the sample-to cample-to-sapple sample correspondence, and collecting the interpolations.
2 Interp(=
=k #1/m+1, A) = (1- 2) + A Vagim. E Sth.
4.2 MODELING EDG WITH STOCHASTIC DIFFERENTIAL EQUATIONS
The continuous trajectory in Section 4.1 significantly enhances the capability to capture evolving patterns, but existing EDG methods can not handle the continuous timestamp data. Hence,
4

---

Published as a conference paper at ICLR 2024
propose to model the data of EDG in the representation space with neural SDEs, since Neural SDEs naturally model continuous temporal trajectories. In contrast, traditional temporal models such as LSTM (Hochreiter & Schmidhuber 1997) and Markov models (Bishop & Nasrabadi 2006) are only able to model discrete timestamps.
Here, SDE-EDG learns the temporal dynamics governing the semantic conditional distributions D(2ly, t) over time. Specifically, SDE-EDG models the temporal trajectory of the data point from the domain at im the arbitrary future timestamp L'm lime Lm of each class k (I....,K]:
L(ne gb (.
ak 2mm
(6)
where the latent variable sn is transformed from m-th domains latent variable 2 and fu is the drift function of the k-th class to capture the evolving patterns, and g& is the diffusion function of the k-th class to characterize the stochastics of the latent representations. Note that is the latent variable (representation) induced by Hence, SDE-EDG can generate the latent continuous trajectory by gradually transforming the sample representation from the current timestamp to any desired future timestamp m/. Thereby, our latent trajectories of SDE-EDG could effectively align with the collected continuous trajectories IFGET, which prevents overfitting to sparse timestamps.
((z), but 2 is the synthetic feature generated by Eq. (6).
We design two objective functions to learn the drift functions f {ff) and diffusion functions K g {gx k= characterized by neural networks: one is aimed to impose Path Alignment Loss in Eq. and another one is downstream classification loss in Eq. (TO). By jointly optimizing {o,
w.r.t these two losses, our approach achieves improved performance on EDG.
4.3 ALIGN SDE-EDG WITH IFGET VIA MAXIMUM LIKELIHOOD
Neural SDEs are designed to capture the dynamics and evolution of data over time, particu- larly in continuous spaces. To the SDE-EDG into the evolving stochastic path given obser- vations, we propose the path alignment regularizer by maximizing its likelihood of the IFGET {2t ck 2t 1.
Simm+> 21m+11m=1.
K Nn Jule ΣΣΣ m= H
M
DoogD(2 2k 2/mm+11== =him) + log D(= = 2/ 2fm++\>> = =dim))
MKNB
Filling the gap between domains with continuous- interpolated samples results in continuous and smooth evolving trajectory over time. Taking Jmte as a regularizer brings two advantages to EDG model training: 1) Empirically, the training process of neural SDEs converges faster with Tmlle as shown in Figure 4a 2) Jmie regularizes the latent space capture evolving patterns. This contributes to learning the evolving patterns the EDG problem and improves the generalization capability to target domains as shown in Figure 2 On the other hand, in the absence of Jmies the model learns invariant representations across domains, leading to the occurrence of the Neural Collapse phenomenon (Han et al.) 2002),, where the latent representations of the same class across the domain collapse to a single point. Consequently, no evolving patterns manifest in the latent representation space shown in Figure 3
Figure 2: the Circle dataset through the SDE-EDG and IRM by Φ. Distinct classes are distinguished by different shapes (triangles and circles), while various domains are denoted by different colors as indicated by the rainbow bar. SDE-EDG successfully learns representations with a discernible decision boundary, whereas IRM collapses towards a single direction, failing depict clear decision boundary.
The left and right images depict representations acquired for
5

---

Published as a conference paper at ICLR 2024
(a) Ground truth
(b) ERM
(c) SDE-EDG
(d) Paths w/ Tmie
(c) Paths w/o Tmie
Figure 3: (a) Ground truth of the Sine dataset, and positive and negative labels red and blue dots separately. (b-c) show prediction results to Sine made by ERM, and SDE-EDG respectively, positive negative predictions are red and blue dots separately. (d) Visualized learned evolving paths (synthetic latent variables) with the Path Alignment loss Imte by SDE-EDG learned from the Sine dataset. (e) Synthetic latent variables without the Path Alignment loss Imic by SDE-EDG learned from the Sine dataset. With Jntes the latent evolving dynamics can be correctly characterized, and SDE-EDO EDDG can capture the evolving patterns.
4.4 SDE-EDG FOR THE PREDICTION LOSS
In this section, we formulate our approach for handling downstream classification tasks. With the Bayes rule, the predictive distribution is
t..) D klt tm Dizly- k fm) D k' b=1t=th tm)
D(y = k|2, 1 = (m) _
(8)
where model (zlly net with input as timestamp i, function denoted as r()). In each iteration, we first compute label distribution with respect time D(ylt = fm) [D(y 1lt fm), ................................ D(y K|t fm)]
k,t tm) with non-pann con-parametric model, and Dlylt tm) neural
Specifically, Dy kelt mm)
ΣB T where S consists of instances sampled from k-th class
18
of m-tt domain, denotes the size of the set. r optimized by minimizing
r (1m Ill
The conditional distribution D(=ly, t) modeled by SDEs lacks an analytic expression, and here we approximate it with a con-parametric model. Given that distributions characterized by SDEs may exhibit either uni-modal or multi-modal patterns, it's also advantageous model multi-modal representations with Neural SDEs (Min et al. 2023). In this context, we present the multi-modal classification loss here but leave the uni-modal loss in the appendix A due to space limitation. To preserve the multi-moda pattern of the latent variables, we employ the non-parametric distribution density method Parzen Window (Parzen 1962)
D(2ly = k, = fm)) Eeeesh
exp( (Dittzz 2))
6 m
where $K includes instances sample from learned SDE-EDG belong to k-th class of m-th domain. By incorporating the estimations of label distribution (D(y (t) and conditional distribution (D(sly, ()), our predictions encompass the temporal evolution of both D(2ly, t) and D(y t).
Model optimization proceeds by minimizing the negative log probability:
Nu M
Jolo ΣΣΣ m=
1 log D(y = k/22 2,, t. tm)
(10)
MKN
The ultimate objective function is Jasa mIie The Maximum Likelihood Loss Jnte is path aligmment regularizer that aims to fit the stochastic evolving paths. The snper-parameter a >> 0 is the weighting of Titl to adjust its contribution the overall loss.
5 EXPERIMENTS
To evaluate the effectiveness SDE-EDG, we verify our method on various datasets, Rotated Gaussian, Sine, and Circle datasets, Rotated MNIST, Portraits, Caltran, Power Supply, and Ocular

---

Published as a conference paper at ICLR 2024
Algorithm 1 SDE-EDG (an iteration during the training phase)
1: Input: S2.... }: . data sets from consecutive domains. Ng: the number of instances sampled for each class in an iteration. R RANDOMSAMPLEI N): a set of N instances sampled uniformly from the set S without replacement. J 0
2: for m in (1,..., M - -o
3: for in {1, .... K) do
4:
3% f RANDOMSAMPLE(S m. NB)
5:
2 2m+1 2m + P.m f& 8) ds + Jem
9h (2t. s)dB, is computed for 2t Sh
6:
S mm+1 f RRANOMSSMMPEE NN:
7:
Use Eq. 4 to search subsequent state 5t *m++1
8:
Use Eq. (5 generate continuous-intee inttmuous-itterpolateedd samples 8m+
9:
I + J + Tmmle where Imle is calculated w.r.t Eq. 77)
for (2,, Y:) S
m do
11:
I J+ I + MKND log D(y = Vel= z4, L = km)
12:
Optimize the loss w.r.t. J
Disease. The objective of this section aims to answer the following key questions (1) What are the evolving trajectories that SDE-EDG learns, and what is the nature of its learning process? (Shown in Figure 3) (2) How does SDE-EDG compare to other methods in improving EDG performance? (such (3) What is the influence of the Maximum Likelihood Loss on the performance of SDE-EDG? (In Figure (d-e) and Figure (c)-(d))
as Table Figure and Figure 4 (a)-(b)
5.1 EXPERIMENTAL SETUP
Rotated Gaussian Zeng al. 2023) consists of 30 domains where each domain has 500 instances generated by the same Gaussian distribution, but the decision boundary rotates from 0" to 338 with interval of 12 We split the domains into source domains (1-22 domains), intermediate domains (22-25 domains), and target domains (26-30 domains). The intermediate domains are utilized the validation set. Circle (Pesaranghader & Viktor 2016) contains evolving 30 domains where the instance are sampled from 30 2D Gaussian distributions. The label is assigned using a half-circle curve as the decision boundary (15 source domains. 5 validation domains, and 10 target domains). Sine (Pesaranghader & Viktor 2016) The label is assigned using a sine curve as the decision boundary. We rearrange this dataset by extending it to 24 evolving domains. (12 source domains, 4 validation domains, and 8 target domains) Rotated MNIST (RMNIST) (RMNIST) (Ghifary et al.) (015)) composed of MNIST digits of various rotations. We follow (Qin et al. (022) and extend it to 30" 19 evolving domains via applying the rotations with degree of (0', 10" ..... 180%) } in order (10 source domains, validation domains, and 6 target domains). Portraits (Ginosar et al. 2015)
Dataset:
(Yearbook (Yo et al. (0222) is real-world dataset that comprises photos of American high school seniors collected over 108 years (1905-2013 for gender classifications. The dataset is divided into 34 domains (19 source domains, 5 validation domains, and 10 target domains). Caltran (Hoffman et al. 2014) consists of real-world images captured by a fixed traffic camera deployed in an intersection over time. We divide it into 34 domains by time. The task of Caltran is to classify scenes identify the presence of one or more vehicles in or approaching the intersection (19 source domains, 5 validation domains, and 10 target domains). PowerSupply (Dau et al.) 2019) is created for the purpose of predicting the current power supply based on hourly records from an Italian electricity company. It includes 30 domains based on days and each data point is labeled as either morning or afternoon (15 source domains, 5 validation domains, and 10 target domains). Ocular Disease (Kaggle) (2020) Ocular Disease Intelligent Recognition (ODIR) set with three classes: Normal, Diahetes and other diseases. Following the EDG setup, sort the photographs in ascending order of the age of the patients (27 source domains, 2 validation domains, and 4 target domains).
Baselines We compare with following baselines: (1) ERM (Vapnik 1999); (2) Mixup Yan et al.
2020); (3) MMD LL et al. 2018b); (4) MLDG (Li et al. 2018a): (5) IRM (Arjovsky et al. 2019] (6)
RSC (Huang et al. 2020); (7) MTL (Blanchard et al. 202 (8) Fish (Shi et al.) 2021): 1b: (9) CORAL
Suun & Saenko| 2016); (10) AndMask (Parascandolo et al. 2020): DIVA (lise et al. 2020); (12)
LSSAE (Qin et al. 2022); ; (13) GI (Nasery al.] 2021a) (14) DDA (Zeng et al. 2023) (15) DRAIN
7

---

Published as a conference paper at ICLR 2024
Table I:
The comparison of the classification accuracy (%) between SDE-EDG and other baseline methods across the synthetic and real-world datasets. The reported results are the average accuracy of the multiple target domains. ("RG" for RotatedGaussian "Cir" for Circle, "RM" for Rotated MNIST, "Por" for Portraits, "Cal" for Caltran, "PS" for PowerSupply, "OD" for OcularDisccse. Gl fails to complete OD due to high time complexity.)
ALGORITHM
RG
CIR
SINE
RM
POR
CAL
PS
OD
AVG
ERM
59.0
49.9
63.0
43.6
87.8
66.3
71.0
57.9
62.3
MIXUP
55.4
48.4
62.9
44.9
87.8
66.0
70.8
59.7
62.0
MMD
56.0
50.7
55.8
44.8
87.3
57.1
70.9
57.6
60.0
MLDG
59.9
50.8
63.2
43.1
88.5
66.2
70.8
43,9
60.8
IRM
47.5
51.3
63.2
39.0
85.4
64.1
70.8
46.2
58.4
DG METHODS
RSC
32.8
48.0
61.5
41.7
87.3
67.0
70.9
54.5
58.0
MTL.
59.0
51.2
62.9
41.7
89.0
68.2
70.7
59.7
62.8
FISH
41.6
48.8
62.3
44.2
88.8
68.6
70.8
48.2
59.2
CORAL
53.0
53.9
51.6
44.5
87.4
65.7
71.0
60.1
60.9
ANDMASK
76.3
47.9
69.3
42.8
70.3
56.9
70.7
51.2
60.7
DIVA
67.9 56.6
52.9
42.7
88.2
69.2
70.8
53.1
62.7
LSSAE
48.7
73.8
71.4
46.4
89.1
70.6
71.1
52.3
65.4
GI
50.8
54.4
65.2
44.6
88.1
70.7
71.4
EDG METHODS
DDA
66.8
51.2
66.6
45.1
87.9
66.1
70.9
55.8
63.8
DRAIN
61.0
50.7
71.3
43.8
89.4
69.0
71.0
58.7
64.4
SDE-EDG
97.7
81.5
72.2
52.6
89.6
71.3
75.7
62.6
75.4
All experimental implementations are conducted using the PyTorch packages and are based on the DomainBed (Gulrajani Lopez-Paz 2020). To ensure a fair comparison, the neural network architecture (shown in Appendix C.2 of the encoding and classification parts are kept constant across all baselines used in different benchmarks. Five independent experiments with different random seeds repeated to reduce the variances.
(Bai et al., 2023
5.2 EXPERIMENTAL RESULTS
SDE-EDG Aligns with the Evolving Trajectories To find out what SDE-EDG learns, we visualize the temporal trajectories sampled by SDE-EDG in Fig 3d It should be noted that SDE-EDG learns the evolving dynamics in the latent space 2, which is not directly interpretable. To address this issue, we use an Identity function as the encoding function, which enables us to learn the dynamics directly raw data space X. Learning in original data space more challenging but provides us with more intuitive understanding of the learned dynamics.
Fig shows that the source domains data are between the range [- 1.200 0). which means we only train machine learning models with the half sine. The trajectories of the same class are compact within this range, and the boundaries between them have large margin to ensure good performance. However, the trajectories become looser and the margins become smaller we move into unobserved domains with timings in the range (0, 31. With a much longer time gap from the source domains, SDE-EDG will eventually fail due to larger discrepancy between learned paths and the ground truth.
Quantitative Results The experimental results of SDE-EDG and other baselines are presented in Table [ which shows the accuracy the average accuracy for all the target domains (complete results for each domain shown in Table 10 average accuracy on all datasets. The results indicate significant improvement over the compared traditional DG methods, which is a reasonable finding since traditional DG methods do not address evolving patterns in EDG. Furthermore, SDE-EDG outperforms LSSAE, DDA, and DRAIN (most recently EDG method) by 10.0%, 11.6%, and 11.0% on overall average accuracy, demonstrating the superior ability of our method to capture evolving patterns. The significant accuracy improvements observed in Table I indicate consistent enhancements in the performance of EDG tasks. In addition, EDG methods perform consistently better than DG methods, showing the importance of modeling evolving patterns to improve prediction performance in EDG tasks. In particular, the OcularDisease dataset is challenging medical image classification task, significantly more complex than standard image classification. SDE-EDG demonstrates its ability to capture evolving patterns in demanding real-world scenarios.
3 for space limitations). SDE-EDG surpasses other baselines'
8

---

Published as a conference paper at ICLR 2024
EAM MUDO
130
KLDG
23
Doauds
Daanda 24
Zee
Donada
235
345
355
385
375
24 23
23
200
611
10
300
200
(a) Training Ace
(b) RMNIST
(c) Cirele
(d) Alation on RM
(c) Alation on PS
Figure 4: Training accuracy convergence trajectories Portraits (b)-(c) Accuracy with the domain index of
RMNIST and Circle Datasets, respectively. (d)-(e) Effects of weighting a on the Maximum Likelihood Loss on RMNIST (RM) and PowerSupply (PS).
Table 2: Rotated MNIST with different temporal gaps At (f here represents domain index).
130"
140
150
160
170
180
AVG
A+
A1/2
75.6 1 0.8
61.8 1 0.8
49.9
1 0.8
50.0 = 0.9
45.1 1 0.7
44.1
44.1 0,9
54.4
A
75.1
0.8
0.9
49.8
4: 0.8
49.8 = 0.8
39.7
0.7
39.7
0.9
52.6
At 2
58.6 - 0.8
0.7
45.6
0.7
42.4 = 0.8
36.9 = 0.8
36.1 1
0.8
44.8
Figure 4bland 4clpot the accuracy trajectory of the baselines (ERM, MLDG, GI) and SDE-EDG across domains for RMNIST and Circle datasets, which show the superiority of SDE-EDG on the other baselines with large margin improvements. SDE-EDG could keep large improvements initially, but eventually, as the distance between the target and source domains increases, all methods will achieve similar performance. Therefore, conclude, that in EDG, models achieve generalization in the relatively near future.
A Ablations: the Impact of Maximum Likelihood Loss on EDG Classifications We conducted an ablation study on the RMNIST and PowerSupply datasets to evaluate the effectiveness of the proposed Maximum Likelihood Loss Juste which aims to train SDE-EDG fit IFGET. Figure 4d and Figure 4e show that SDE-EDG achieves the best performance with dataset and a 10 on the PowerSupply dataset. The above empirical results suggest that Jnte improves the performance of EDG by aligning the underlying evolving paths (optimizing the Path Alignment loss Jpa) and quantifying stochastic uncertainties (minimize Stochastic Uncertainty loss Jou), details proof in Appendix D.I Specifically, when we only apply classification loss with O, the performance is the worst. On the other hand, with much larger a = 200, SDE-EDG focuses on aligning the evolving paths and giving lower importance to classification tasks. Thus, aligning stochastic evolving processes improves performance in EDG.
1 on the RMNIST
Ablations: At influence on EDG performance In Table we set interval At 5 and 20 between source domains with SDE-EDG, where the 10t interval is in the original setting. With smaller At (At x 2 + At > 2t/2), the accuracy experiences consistent improvement, finding that aligns with our motivation: smaller temporal gap between domains reduces the generalization error. Therefore, interpolations between temporal gaps as an approximation to the sample at an arbitrary timestamp would lead to reduced A/ and overcoming overfitting to available limited timestamps.
6 CONCLUSION
This work presents a new approach SDE-EDG for modeling Evolving Domain Generalization (EDG). Our approach involves constructing IFGET by identifying sample-to-sample correspondence and generating continuous-interpolle
samples via linear interpolations. Subsequently, we employ Stochastic Differential Equations (SDE) and train it in alignment with IFGET. Our contribution lies in revealing the importance of capturing the evolving patterns through the collected individual's temporal trajectories, and of interpolating between time intervals to mitigate the issue of the limited number of source timestamps, which effectively prevents SDE-EDG from overfitting the limited timestamps. We also provide a theoretical analysis demonstrating that our method can reduce the generalization risk.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We appreciate constructive feedback from anonymous reviewers and meta-reviewers. This work is supported by the Natural Sciences and Engineering Research Council of Canada (NSERC), Discovery Grants program.
REFERENCES
Spyros Makridakis vangelis Addison Howard inversion. M5 forecasting accuracy, 2020, URL com/ wwwpetitiooeeeeeeeeee m5 forecasting / accuracy
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: met:::/000009000 2019.
Guangji Bai, Chen Ling, and Liang Zhao. Temporal domain generalization with drift aware dynamic neural networks. In The Eleventh International Conference on Learning Representations, 2023.
URL
net het.eeeenneettooorddddddoooottt
Christopher Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume Springer, 2006.
Gilles Blanchard, Aniket Anand Deshmukh, Urun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. The Journal of Machine Learning Research, 22(1):
46-100, 2021.
John Blitzer, Ryan McDonald, and Fernando Pereira. Domain adaptation with structural correspon- dence learning. In Proceedings of the 2006 conference on empirical methods in natural language processing, pp. 120-128, 2006.
Michelle Boué and Paul Dupuis. A variational representation for certain functionals of brownian motion. The Annals Probability, 26(4):1641-165) 1998.
Liang Chen, Yihang Lou, Jianzhong He, Tao Bai, and Minghua Deng. Geometric anchor correspon- dence mining with uncertainty modeling for universal domain adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. (6134-16144 2022.
Qi Chen, Changjian Shui, Ligong Han, and Mario Marchand. On the ptaattity dilemma in continual meta-lcarning Theory and algorithm. Advances Neural Information Processing Systems, 36, 2024.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018a.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018b.
Debasmit Das and CS George Lee. Sample-to cumple-to-sample sample correspondence for unsupervised domain adaptation. Engineering Applications of Artificial Intelligence, 73:80-91, 2018.
Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The uer time series archive. IEEE/CAA Journal of Automatica Sinica, ((6):1293-1)) 1305, 2019.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 411-1222 2012.
Seth R Flaxman, Rupert RA Bourne, Serge Resnikoff, Peter Ackland, Tasance Braithwaite, Maria Cicinelli, Aditi Das, Jost B Jonas, Jill Keeffe, John H Kempen, al. Global causes of blindness and distance vision impairment 1990-2020: a systematic review and meta-analysis. The Lancet Global Health, 5(1 1221-e1221112333 2017.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In Proceedings of the IEEE international conference on computer vision, pp. 2551-2559, 2015.
10

---

Published as a conference paper at ICLR 2024
Shiry Ginosar, Kate Rakelly, Sarah Sachs, Brian Yin, and Alexei A Efros. A century of portraits: A visual historical record of american high school yearbooks. In Proceedings of the IEEE International Conference Computer Vision Workshops, pp. 1-7, 2015.
Hans E Grossmiklaus, John M Nickerson, Henry F Edelhauser, Louise AMK Bergman, and Lennart Berglin. Anatomic alterations in aging and age-related diseases of the eye. Investigative ophthal- mology & visual science, 54(14) (): 2013.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprini arXiv:2007.01434, 2020.
X.Y. Han, Vardan Papyan, and David I.. Donoho. Neural collapse under MSE loss: Proximity and dynamics on the central path. In International Conference on Learning Representations, 2022.
URL URL htt.hhtts::///t /tti:p:://operreeee
www.neeeooormdaaaaeeme 1bdvwH R3
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Judy Hoffman, Trevor Darrell, and Kate Saenko. Continuous manifold based adaptation for evolving visual domains. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 867-874, 2014.
Zeyi Huang. Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves rross-domain generalization. In Computer Vision-ECCI 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 124-140. Springer, 2020.
Maximilian Ilse, Jakub M Tomczak, Christos Louizos, and Max Welling, Diva: Domain invariant variational autoencoders. In Medical Imaging with Deep Learning, pp. 322-348. PMLR, 2020.
Kaggle. Kaggle: ocular disease recognition.
https://wwe hatp:://www.aggttt kaggle com/ haoo//adreeea andrewmvdl 2020. Accessed: 2022-06-15.
Patrick Kidger, James Foster, Xuechen Li, and Terry J Lyons. Neural sdes as anfiite-dimensionn gans. In International Conference on Machine Learning, pp. 5453-5463. PMLR, 2021.
Matthicu Kirchmeyer, Alain Rakotomamonjy, Emmanuel de Bezenac, and patrick gallinari. Mapping conditional distributions for domain adaptation under generalized target shift. In International Conference on Learning Representations, 2022. httpsttp:: /tttps:///ppnrevlll net/ forum?
[d=aP[32P187B2
Lingkai Kong, Jimeng Sun, and Chao Zhang. Sde-net:
Equipping deep neural networks with uncertainty estimates. In International Conference on Machine Learning, pp. 5405-5415. PMLR, 2020.
Atsutoshi Kumagai and Tomoharu Iwata. Learning non-linear dynamics of decision boundaries for maintaining classification performance. In Proceedings of the AAAI Conference Artificial Intelligence, volume 31, 2017.
Ananya Kumar, Tengyu Ma, and Percy Liang. Understanding self-training for gradual domain adaptation. In International Conference on Machine Leurning, pp. 5468-5479. PMLR, 2020.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta-learning for domain generalization. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018a
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex Kot. Domain generalization with adversarial feature learning. In Proceedings the IEEE conference computer vision and pattern recognition, pp. 5400-5405 2018b.
Xuechen Li, Ting- Kam Leonard Wong, Ricky TQ Chen, and David Duvenaud. Scalable gradients for stochastic differential equations. In International Conference on Artificial Intelligence and Statistics, pp. 3870-3882, PMLR, 2020.
11

---

Published as a conference paper at ICLR 2024
Hong Liu, Mingsheng Long, Jianmin Wang, and Yu Wang. Learning to adapt to evolving domains. Advances in neural information processing systems, 33:22338- 223488, 2020a.
Xuanqing Liu, Tesi Xiao, Si Si, Qin Cao, Sanjiv Kumar, and Cho-Jui Hsieh. How does noise help robustness? explanation and exploration under the neural sde framework. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 282-290, 2020b.
Xuanchen Lu, Xiaolong Wang, and Judith E Fan. Learning dense correspondences between photos and sketches. 2023.
Massimiliano Mancini, Samuel Rota Bulo, Barbara Caputo, and Elisa Ricei, Adagraph: Unifying predictive and continuous domain adaptation through graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6568-6577, 2019.
Zelda Mariet and Vitaly Kuznetsov. Foundations of sequence-to-ssquueec modeling for time series. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), Proceedings of the Twenty-Second Interna- tional Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine
Learning Research, pp. 408-417. PMLR, 16-18 Apr 2019.
Ming Min, Ruimeng Hu, and Tomoyuki Ichiba. Directed chain generative adversarial networks. arXiv preprint arXiv:2304. 20t.:33344.1131,, 2023.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International conference machine learning. pp. 10-18. PMLR, 2013.
Anshul Nasery, Soumyadeep Thakur, Vihari Piratla, Abir De, and Sunita Sarawagi. Training for the future: A simple gradient interpolation loss to generalize along time. Advances in Neural Information Processing Systems, 34: 14:19198-19209 2021a.
Anshul Nasery, Soumyadeep Thakur, Vihari Piratla, Abir De, and Sunita Sarawagi. Training for the future: A simple gradient interpolation loss to generalize along time. Advances in Neural Information Processing Systems, 34: 14:19998-1,909, 2021b.
A Tuan Nguyen, Toan Tran, Yarin Gal, and Atilim Gunes Baydin. Domain invariant representation learning with domain density transformations, Advances in Neural Information Processing Systems, 14:5264-5275. 2021.
Bernt Øksendal and Bernt Øksendal. Stochastic differential equations. Springer, 2003.
Giambattista Parascandolo, Alexander Neitz, Antonio Orvieto, Luigi Gresele, and Bernhard Schölkopf. Learning explanations that are hard vary. arXiv preprint arXiv:2009.00329, 2020.
Sung Woo Park, Dong Wook Shu, and Junseok Kwon. Generative adversarial networks for markovian temporal dynamics: Stochastic continuous data generation. In International Conference Machine Learning, pp. 8413-8421, PMLR, 2021a.
Sunghyun Park, Kangyeol Kim, Junsoo Lee, Jacgul Choo, Joonseok Lee, Sookyung Kim, and Edward Choi. Vid-ode: Continuous-time video generation with neural ordinary differential equation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 2412-2422, 2021
Emanuel Parzen. On estimation of a probability density function and mode. The annals of mathemat- ical statistics, 3363):006 1962.
Ali Pesaranghader and Herna L Viktor. Fast hoeffding drift detection method for evolving data streams. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECMI. PKDD 2016, Riva del Garda, Italy, September 19-23, 2016, Proceedings, Part II 16, pp- 96-111. Springer, 2016.
Tiexin Qin, Shigi Wang. and Haoliang Li. Generalizing evolving domains with latent structure- aware sequential autoencoder. In International Conference on Machine Learning, pp. 18062-18082. PMLR, 2022.
12

---

Published as a conference paper at ICLR 2024
Tom Ryder, Andrew Golightly, A Stephen McGough, and Dennis Prangle. Black-box variational inference for stochastic differential equations. In International Conference on Machine Learning. pp. 4423 4432. PMLR, 2018.
Shiori Sagawa, Pang Wei Koh, Tatsumori Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference Learning Representations.
Shiori Sagawa, Pang Wei Koh, Tatsumori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arviv: 191 1.08311 2019.
Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. arXiv preprint arXiv:2104.09937) 2021.
Changjian Shui, Qi Chen, Jun Wen, Fan Zhou, Christian Gagné, and Boyu Wang. A novel domain adaptation theory with jensen-shannon divergence. Knowledge-Basel Systems, 257: 2022. ISSN 0950-7051.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13455 2020.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision-ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16. 2016, Proceedings, Part III 14, pp. 443-450 Springer, 2016.
Yifan Sun, Linan Zhang, and Hayden Schaeffer. Neupde; differential equations for modeling time-dependent data. In Mathematical and Scientific Machine Learning, pp. 352-372. PMLR, 2020.
Neural network based ordinary and partial
Vladimir N Vapnik. An overview of statistical learning theory. IEEE transactions on neural networks, 10(5):988-995 1999.
Martin J Wainwright. High- digeeddimnsional statistics: non-asymptotic viewpoint, volume 48. Cam- bridge University Press, 2019.
Hao Wang, Hao He, and Dina Katabi. Continuously indexed domain adaptation. arXiv preprint arXiv:2007.01 807, 2020.
Haoxiang Wang, Bo Li, and Han Zhao. Understanding gradual domain adaptation: Improved analysis, optimal path and beyond. In International Conference on Machine Learning, pp. 22784-22801 PMLR, 2022.
Winnie Xu, Ricky TQ Chen, Xuechen Li, and David Duvenaud. Infinitely deep bayesian neural net- works with stochastic differential equations. In International Conference on Artificial Intelligence and Statistics, pp. 721-738. PMLR, 2022.
Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprint arXiv:2001.00677, 2020.
Huaxiu Yao, Caroline Choi, Yoonho Lee, Pang Wei Koh, and Chelsea Finn. Wild-time: benchmark of wn-the-wild distribution shift over time. In ICML 2022 Shift Happens Workshop, 2022. URL httpp:://ppeneeieew.neee.
Hsiang-Fu Yu, Nikhil Rao, and Inderjit S Dhillon. Temporal regularized matrix factorization for high-dimensional time series prediction. Advances in neural information processing systems, 29, 2016.
Qiuhao Zeng, Wei Wang, Fan Zhou, Charles Ling, and Boyu Wang. Foresee what you will lear: Data augmentation for domain generalization in non-stationary environment. Proceedings of the AAAI Conference on Artificial Intelligence. 379) Jun. 2023. doi: 10. 10..160//aai.v3377 26320. http:p//....aaaaaa.......//.a/aa |tt.a///a//////////a/o///o/ooo..
13

---

Published as a conference paper at ICLR 2024
Qiuhao Zeng, Wei Wang, Fan Zhou, Gezheng Xu, Ruizhi Pu, Changjian Shui, Christian Gagne, Shichun Yang. Boyu Wang, and Charles Ling. Generalizing across temporal domains with koopman operators. arXiv preprint 240.v:240..07834, 2024.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018. URL https: //openreview. net/ forum ? 1d=r1bap1-R8
14

---

Published as a conference paper at ICLR 2024
A UNI-MODAL CLASSIFICATION LOSS
We calculate the Evolving Centroid of domain at bm the mean vector of the instances belonging to Sk 26 " o No E(eg
gk (8 of 1 3//m* where $ be the set of NB data points sampled from D,.. m
(short for D(2, y (lm) in training iteration by SDE-EDG, i/mm stands for the i-th instance time bm. and Ng B is the number of instances sampled in a training iteration belonging to k-th class of the domain at timestamp mm-
At each timestamp, the Evolving Centroids exhibit holistic evolving patterns reflecting the dynamics of class-conditional distribution. This differs from the conventional Domain Adaptation (DA) or Domain Generalization (DG) methods, which aim to learn the invariant features by maintaining global centroid for each class(Arjovsky et al. 2019 Li et al. 20186 Muandet et al. 2013; Nguyen al. 2021). Instead, we do not aim ahgn the class centroid across all domans but allow the Evolving Centroids to evolve over time, which offers a more accurate representation of the data distribution's temporal changes. The predictive distribution for Dn is
D(y kle,t tm) exp( exp(-Dist(2, (.))
(11)
exp( Dist(z, (
During the training stage, at each step, we sample NB samples from each class l in Sm, which used to compute Evolving Centroid ex to make predictions for data in Sm-
In the inference stage, we search the closest Evolving Centroids sampled by SDE-EDG at given timestamp, and assign the class of the closest Evolving Centroid as the classification prediction.
B FULL EXPERIMENTAL RESULTS
In this section, we present complete experimental results to validate the efficacy of our proposed evolving domain generalization task. Our findings demonstrate that, across wide range of scenarios, our approach consistently outperforms existing domain generalization baselines, thus establishing new kate-of-the-art performance on existing benchmarks. These results affirm the effectiveness of our method in addressing the challenges posed by evolving domains. By achieving superior performance, our approach shows applicability to real-world problems.
Table 3: Rotated Gaussian. We show the results on each target domain by domain index.
ALGORITHM
26
27
28
29
30
AvG
ERM
58.8 + 1.3
58.0 +
1.5
57.8
1.3
62.0
1.1
58.6 +
1.1
59.0
MIXUP
56.2
1.5
63.4
3.0
56.8
1.5
4.0
55.4
MMD
0.9
53.0
1.0
1.6
63.6
56.0
MLDG
1.6
5..8
57.2+
57.2
62.2
1.3
59.9
IRM
1.9
5..8
51.8
41.6
31.4
47.5
RSC
1.9
3.2
31.8
3.0
3.0
32.8
MTL
2.2
5.
58.8
59.8
67.6
59.0
FISH
52.0
49.6
44.6.1..0
35.6
26.2
41.6
CORAL
1.
54.0
52.0 53.8
0.8
50.6
53.0
ANDMASK
81.8
1.64
7..4
75.4
69.4 76.3
DIVA
59.0 1.5
53.6 55
59.2
55.6
1.5
56.6
LSSAE
0.9
50,8
43.4
2.44
50.4
48.7
GI
51.6
53.2
47.4
49.6
52.4
+ 2.0
50.8
DDA
1.3
75.0 1.5
68.2
1.1 60.8 1.0
52.8
+ 1.5
66.8
DRAIN
73.2 13.2+2.9
70.0 1.7
63.8
2.4
53.2
45.0
1.2
61.0
SDE-ED EDG (OURS)
98.4 + 0.6
97.0
97.4
0.8
97.8
0.6
97.8
0.7
97.7
15

---

Published as a conference paper at ICLR 2024
Table 4: Circle.
We show the results each target domain by domain index.
ALGORITHM
21
23
25
26
25
29
30
Avri
FRM
53.9 1.9+-.5
55.8 + 4.8
$3.9 11.9-5 5.2
44.7
6
56.9
8
41.9
7.5
41.7
5.9
$4.2
300
47.8
T
49
48.6=3.8
41.7
$4.2
MMD
57.7
47.
41.7
IRM
1.99
59.4
$6.9
48.1
57.5
43.8
11.9
7.5
41.7
5.9
54.
4.3
51.3
RSC
15.31.366
$14138 3.
19.14
43.158
56.9
44
47.8
41.9
1.5
41.7
11,7.15
54.2
3.0
47.8 5.7
48.0
MIL
21.4
57.2
53.3
48.3
36.9
4.8
49.2
43.3
5.0
45.3
23
54.2
5.7
42.2
51.2
FISH
23,
53.1
49.4 4.5
47.8
41.7
54.2
47.3
48.8
CORAL
ANDMASK
DIVA
$1.3
76.3 4.2
74.7
56.7 5.1
67.0
6.1
62.3
5.1
62.0
5.6
66.3
4,1
70.3
5.6
62.0
67.9
LSSAE
95.3 1.9
95.6 2.1
93.5
299
96.1
81.8
42
74.1
$1.9
5.6
$2.3
46.5
9.2
48.4
5.3
738
GI
72.0
620
61.0
58. 28
56.0
49
490 0.7
42.0
1.4
45.0 21
55.0
3.5
44.0 0.0
54.
DDA
7..0
5000
51.0
440 28
55.0
0
49.0
42.0
45.0
55.0
3.5
44.0
4.2
$1
DRAIN
48.0+2.11
(OURS)
Table 5: Sine.
We show the results each target domain by domain index.
ALGORITHM
17
18
19
20
21
22
23
24
AvG
ERM
71.4 1 6.1
91.0
81.6
2.4
53.4
2.9
$1.1
6.7
54.3
4.7
49.5
4.8
$1.7
1.7715.0 5.0
63.0
MIXUP
93.5
MMD
57.1 57.0 4.2
4
5.4
50.0
$5.1260
54,4
4.7
49.5
4.8
51.7
5
55.8
MI.GG
69.2
4:17
52.1
+ 5.4
50.7
1.8
51.1
6.7
4.73
49.5
4.8
51.7
$1.74:50
5.0
63.2
IRM
6.2
81.
56.6
6.0
5.82
5.17
49.5
51.7
63.2
RSC
6.6
83.5
1.9
84.5
$2.8
2.8
55.1
$..4
4.8
$1.7
61.5
7
62.9
FISH
66.1
82.0
87.5
1 2.4
55.2
1 3.0
$1.1 1 6.7
54.3
4.7
49.5
1 4.8
$1.7
11.7..51 5.0
62.3
CORAL
6..0
4..1
48.6 6.4
50.7
+ 1.8
4.2
48.6
4.0
5.07
51.6
ANDMASK
5.1
42,9
54.2
71.9
1.9
86.4
3.2
90.4
88.1
3.4
76.4
69.3
DIVA
6.6
60.8
47.6
2.6
50.0
2.8
55.1
+ 6.7
$1.9
38.6 4.8
4.4
$2.9
LSSAE
93.0
0.7
69.2
6.8
68.8
4.
63.9
4..0
71.4
GI
0.7
4.42
89.1
60.9 3.6
55.1
4 0.8
53.5 6.0
49.8
198444 4.1
52.0
65.2
DDA
0.7
0.6
74.2
1.6
89.6
75.5
70.0
59.1
74.0
66.6
DRAIN
4.9
44,6
69.3
89.6
82.8
78.5
70.2
9..0
71.3
SDE-EDG (OURS)
99.0
0.6
90.0
88.8
1
0.6
64.9
1
$2.4
12.403
43.0
4.9
42.3
4.0
72.2
Table RMNIST. We show the results on each target domain by domain index.
ALGORITHM
130"
140"
150"
160
170"
180"
ERM
56.8 0.9
44.2
0.8
37.8
0.6
38.3 + 0.8
40.9 0.8
43.6
0.8
43.6
MIXUP
6.7
47.4
39.1
38.3 0.7
40.5
42.8
0.9
MMD
5.9
4.80
3.7
4..6 0.8
43.7
0.8
44.8
MLDG
5.74
4.9
3.8
3..
39.9
42.0
43.1
IRM
4.9
3.7
3.7
35.7
37.8
40.3
39.0
RSC
5..1
4..9
3.7
3..0
3.8
41.6
41.7
MTL
5.9
4..1
3..4
36.1
39.1
4.89
41.7
FISH
60.8
4..8
37.6 3.82
39.0
0.8
40.7
0.7
44.2
CORAL
58.8
4.8
38.5 3.79
41.3
0.8
43.5
44.5
ANDMASK
5.9
4.89
0.7
38.6 0.8
40.8
0.8
43.2
33.2008..
42.8
DIVA
5.8
4..0
3..6
3..9
3.8
40.1 42.7
LSSAE
6.81
51.6
4.84
3.76
40.3
4..4
46.4
GI
6..6
4.
0.8
40.0
0.8
40.1
40.1
44.6
DDA
60.7
5..0
42.6
39.6 0.8
38.0
39.7
+ 0.8
45.1
DRAIN
59.5
4..4
40.2
37.2 0.7
39.6 0.8
11.0
0.7
43.8
SDE-EDG (OURS)
75.1
49.8 0.9
0.8
+ 0.8
39.7 0.7
39.7
+ 0.9
52.6
16

---

Published as a conference paper at ICLR 2024
Table 7: Portraits. show results on each target domain by domain index.
ALGORITHM
25
26
27
28
29
30
31
32
34
Avri
FRM
75.5 0.9
83.8 + 0.9
88.5
0.8
93.3
91.4
92.1
0.7
90.6 + 0.8
$4.3
0.9
58
0.9
9 +
87
0.9
88.5
84.3
87
MMD
85.5
90.1
94.39444444
94
85.6
67
IRM
74,2 0.9
83.5
88.5
9100 0.8
90.4
0.7
87.3
87.0
0.9
30.4
86.7
0.9
85.1
83.8
RSC
1.99
84.7
0.8
87.9
0.7
93.3
0.7
92.5
007
91.0
0.7
90.0
1
0.7
$4.6
1.2
88.2
0.8
85.8
87.3
MIL
78.2 =:9
$6.3
= 0.3
90.9
0.8
94.2
93.8
0.
92.0
0.7
91.2
= 0.7
56.0
1.2
59.3
0.8
87.4
89.0
FISH
0.9
86.9
89.5
93.5
92.1
36.2
1.3
58.7
87.7
88.8
CORAL
ANDMASK
70.
75.2
74.1
64.7
77.3
DIVA
26.2
86.6
88.3
4.8
93.5
0.7
93.1
1 0.5
91.6
91.1
0.7
$4.1
89.1
0.8
87.0
1.5
88.2
LSSAE
77.7 0.9
87.1
0.8
90.5
+
0.7
94.1
0.6
94.3
D.5
92.2
0.6
91.2
0.7
56.7
89.6
U.S
86.9
89.1
GI
77.3 1.2
86.6
1.3
90.8 11
95.3
93.1
893
1.
84.1
1.
N7.7
10
87.5
2.0
88.1
DDA
76.0
85.6
886
936
92.9
0
92.9 0.6
90.3
84.3
1.2
58
85.9
2 1.2
87.9
DRAIN
(OURS)
86.6
93.3
Table 8: Caltran. We show the results each target domain domain index.
ALGORITHM
25
27
AVG
ERM
29.9
3.5
88
61
1.5
56 16.311
90.0
1.6
60.1
= 2.5
55.5
+ 3.5
88.8
2.4
5T
3.5
50.5
5
+
66
MIXUP
53.6
3.9
8900
+
2.0
61.8
+
24
55.7
88.2
2.1
58.6
3.0
52.3
3,7
88.6
2.7
57
3.0
55
4.3
66
MMD
30.2 1.1
92.7
1.7
56.4
3.7
39
52.1
3.2
42.8
3.0
92.1
2.2
42.1
29.4
3.8
$7.1
MLDG
IRM
90.8
90.9
RSC
53.0
88.4
62.6
1.0
56.5
88.0
14
39.4
$1.9
29
90.0
2.0
59.4
1.J
67.0
MTL
64.2
87.2
64.
3.9
60.0
84.3
+2
60.6
52.6
3.7
83.9
29
$8.2
4.1
+3.7
68.2
PISH
31.1
88.2
1.5
64.7
4.0
$7.9 3.1
38.3
22
59.9
57.5
2.7
87.4
2.
57.7
3.7
6..
68.5
CORAL
3..0
90.3
61.2 3.8
15.0
56.8
188
$2.0
1.6
5..9
63.7
ANDMASK
30.0
56.2
42.6
41.2
DIVA
90.1
58.7
62
69.2
LSSAR
62.6
58
62.0
92.
60.5
70.6
GI
+31
86.6
19
65.5
12
60.641
88.8
58.5
3.6
53.1
28
88.7
61
78.0
70.7
DDA
31.0 11.01334 3.4
92.6
56.8
0.9
59
94.0
22
61.
2.3
52.9
23
92.9
S1.8
5.
62.9
5.8
66
DRAIN
66.4
+ 3.3
8
+
10
65
22
628112
779
15
62.1
55.7
+ 37
7899
68.6
38
75.7
56
69.0
SDE-EDG
2.9
88.8
55.1
89.5
88.2
72.2
71.3
Table 9: PowerSupply. We show the results on each target domain by domain index.
23
ERM
69.3
70.0
64.4
76.0
1.
70
69.8
69.0
65.5
71.0
MIXUP
69.6 1.4
69.5
+
1.5
6
64.1
87
1.0
76.6
1.3
70.1
1.4
69.2
68
65.0 +
70
MMD
70.0
69.7
1
1.4
68.7
14
6488
85.6
L.
76.1
1.3
10.0
1.5
69.5
14
68.7
1.
65.6
1
70.9
MLDG
19.711.4
69.7
1
68.6
61.
86.4
L1
76.3
1.4
70.1
69.4
1.3
68
65.6
1.5
70.8
IKM
1.4
68.3
0.3
76.3
68.2
20.8
16.3
MTL
69.6
69.4
68.2
64.2
87
76.6
69.9
1,3 69.1
682
64.6
70.7
FISH
1.4
69.4
1.4
68.2
1.4
64.2
2
81.3
L8
76.6
= 1.3
69.9
1.5
09.1
68.2
1.3
63.2
1.5
70.8
CORAL
69.9
69.7
1
L4
68.9
14 14
64.6
86.1
LG
76.3
1.3
70.0
1.5
69.3
00.3:11
68.3
1.3
1
71.0
ANDMASK
6..9
69.4
68.2
64.0
87 +.9
76.7
=
70.0
1.5 69.1
68.0
1.3
64.7 1.5
70.7
72.9
72.0
71.8
DDA
69.8
72.4
70.5
638
83.7
L2
73
70.1
71.4
70.5
63.4
1.2
70.9
DRAIN
70.1
2:
70.0
1.
69.3
65.5
83
758
70.3
1.3
69.8
68.9
9
66
71.0
SDE-EDG (OURS)
67.7
1.2
74.2
1.5
1.
75.2
87.9
1
LG
75.0
67.2
1.6
72.0
1.5
79.
1.2
75.3
L.
75.7
C EXPERIMENTAL SETUP
C.I DATASETS
Rotated Gaussian (Zeng et al.) 2023) consists of 30 domains generated by the same Gaussian distribution, but the decision boundary rotates from 0' to 338 with an interval 12", We split the domains into source domains (1-22 domains), intermediate domains (22-25 domains), and target domains (26-30 domains). The intermediate domains are utilized as the validation set.
Dataset:
Dataset: Circle (Pesaranghader & Viktor 2016) contains evolving 30 domains where the instance are sampled from 30 2D Gaussian distributions. The label is assigned using a half-cirele curve as the decision boundary, (15 source domains, 5 validation domains, and 10 target domains)
17

---

Published as a conference paper at ICLR 2024
Table 10: OcularDisease, We show the results on each target domain by domain index.
30
31
AvG
ERM
53.4 1 5.8
58.1
+ 4.8
60.8 1: 5.6
59.2
1
7.4
57.9
MIXUP
99.1+4.8
67.8
+ 2.9
46.7 + 4.6
65.4 + 5.8
59.7
MMD
58.0
7.0
66.7
49.6 7.2
57.6
MLDG
4..61.6.0
57.8
3.7
41.21
43.9
IRM
3.2
4.3
6.8
50.4
46.2
RSC
4.72
4.74
49.2
7.7
54.5
MTL
58.0
6.3
56.7 14.1
62.1
59.7
FISH
5.2
4..6
4.3
6..0
48.2
CORAL
7.0
6..6
54.2
6.54
60.1
ANDMASK
7.1
6.
5.10
4.7
51.2
DIVA
5.11
5.2
51.7
56.7
53.1
LSSAE
4.80
45.3
$7.5
5.92
52.3
DDA
1.94
63.6 0..6331
44.2
1421600
62.1
02.1511
55.8
DRAIN
58.0
62.8
6.4
53.3 6.8
60.8 + 9.8
58.7
SDE-EDG (OURS)
6.7
66.4 5.9
64.2 8.8
65.4 8.7
62.6
Table 11: Neural network architectures different datasets. (MLP is short for Multiple-layer Perceptrons)
DATASET
FEATURE EXTRACTOR
CLASSIFIER
f &
RGAUSSIAN
2,32, 32 MLP 2,32, 32 -MLP
A LINEAR LAYER
32, 32) -MLP
CIRCLE
A LINEAR LAYER
32,32 -MLP
SINE
2,322, 32).32]-1 -LLP
A LINEAR LAYER
32,32 32. -MLP
ROTATING MNIST
MNIST CONVNET RESNET-18
A LINEAR LAYER
512.512,512)
PORTRAIT
A LINEAR LAYER
(12]511
CALTRAN
RESNET-18
A LINEAR LAYER
512, 512 -MLP
POWERSUPPLY
[2, 256, 256--M5P
A LINEAR LAYER
256,256 256 -MLP
OCULARDISEASE
RESNET-
A LINEAR LAYER
512,512 MLP
Sine In Sine (Pesaranghader & Viktor, 2016) each data owns two attributes (x), The label assigned using a sine curve as the decision boundary. We rearrange this dataset by extending it to 24 evolving domains. Each domain covers 24 the period of the sinusoid. (12 source domains, validation domains, and 8 target domains)
Dataset:
Dataset: Rotated MNIST (RMNIST) (Ghifary al. 2015) is adaptation of the popular MNIST digit dataset (Degg 2012), composed of MNIST digits of various rotations. The task is to classify digit from 0 to 9 given an image of the digit. We follow (Qin al. 2022) and extend it to 19 evolving domains via applying the rotations with degree of (0", 15" 30" 180%) } in order. (10 source domains, 3 validation domains, and 6 target domains).
Dataset: Portraits GGnossr et al. 2015) is a real-world dataset that comprises photographs of
American high school seniors collected over period of 108 years (1905-2013) across 26 states. The objective is to accurately classify the gender for each photograph. The dataset is divided into 34 domains based fixed interval over time. (19 source domains, validation domains, and 10 target domains)
Dataset: Caltran (Hoffman et al., 2014) consists of real-world images captured by fixed traffic camera deployed in an intersection over time. Frames were updated at 3-minute intervals each with resolution 320 x 320. We divide into 34 domains by time. The task of Caltran to classify scenes to identify the presence of one raise from the continually evolving domain shift as changes include time, illumination, weather, etc. (19 source domains, 5 validation domains, and 10 target domains)
vehicles in or approaching the intersection. The challenge mainly
Dataset: PowerSupply (au et al. 2019) is a dataset designed for the task of time-section prediction of current power supply based on hourly records obtained from Italian electricity company. The dataset consists 30 domains formed according to days. Each data point is assigned binary class label indicating whether the current power supply belongs to the morning or the afternoon. Domain
18

---

Published as a conference paper at ICLR 2024
Table 12: Hyper- parametes selected values
DATASET
PARAMETERS
VALUE
&
10
RGAUSSIAN
LEARNING RATE
le-3
BATCH SIZE
64
CIRCLE
LEARNING RATE
le-3
BATCH SIZE
64
10
SINE
LEARNING RATE
le-3
BATCH SIZE
RMNIST
LEARNING RATE
le-3
BATCH SIZE
48
&
PORTRAITS
LEARNING RATE
le-4
BATCH SIZE
24
CALTRAN
LEARNING RATE
5e-5
BATCH SIZE
24
10
POWERSUPPLY
LEARNING RATE
le-3
BATCH SIZE
64
0.5
OCUlARDiSEASE LEARNING
Se-5
BATCH SIZE
24
shifts may arise due to variations in season, weather, price, or the differences between working days and weekends. (15 source domains, 5 validation domains, and 10 target domains)
Dataset: OcularDisease (Kaggle, 2020) (from the Kaggle Competition (Kaggle 2020)) Ocular Disease Intelligent Recognition (ODIR) a structured ophthalmic database of 5,000 patients with age, color fundus photographs from left and right eyes and doctors' diagnostic keywords from doctors. We set three classes: Normal, Diabetes and other diseases. generate non-stationary environments, sort the photographs in ascending order of the age of the patients. (27 source domains, 2 validation domains, and 4 target domains)
C.2 EXPERIMENT SETTING AND IMPLEMENTATION DETAILS
Neural network architectures used for different datasets in Table II ResNet18, and MNIST ConvNet are from domainbed codes (Gulrajani & Lopez-Pat 2020). Specifically, Table demonstrates the classifier architecture for baselines equipped with learned classifier. SDE-EDG does not equip with classifier network, since the prediction is based on the closest distance to the Evolving Centroids.
We list the values of the torchsde (Kidger et al. 2021) package for SDE implementations and the step size is 0.05.
of SDE-EDG for different datasets in Table 12 We
For SDE-EDG, to address the issue of lacking access data ln. adopt sampling strategy that z is sampled from N (pt. ool) using the Reparameterizztion trick, where μ is a learned parameter and dg is predefined constant.
D THEORETICAL ANALYSIS
In this section, show Jnte can be decomposed into sums of Path Alignment Loss for stochas- tie path fittings, and Stochastic Uncertainty Loss for minimizing diffusion terms at observations. Furthermore, we prove that SDE-EDG learning methodology leads lower generalization bound. Lemma D.I. The optimization loss Tmte defined in Eq. (7), is of the upper bound of the Path Alignment loss Jya and the Stochastie Uncertainty loss Jau. which implies (the proof in Appendix E.5
19

---

Published as a conference paper at ICLR 2024
Jmll
M
K
NB
(2m041
2m g(2m) )orm)22m )2
M
NB
1
(og(22gg(2 (2m) )
2MKNB
2MK
Jem
where hide the subscript, sample index for Zmal 2m for simplicity.
Lemma D.I shows (1) The term Jpa is included in Eq. (3) Thus, minimizing Jpa can lead to
smaller ( and eventually help to minimize the expected generalization error Eq. [12). (2)
The Stochastic Uncertainty loss J... serves domains at timestamps {4,,..., bm...... ,.,,...,, (M are observed, the stochastic uncertainty depicted by
loss function to optimize the diffusion terms As
diffusion term at these timestamps should be small (Kong et al. (2020).
Remark D.2. (adapted from (Shui et al. 2022)) Let be the stochastic path of unseen target evolving domains sampled sequentially from an evolving environment between timestamps from to T T", and be the learned stochastic path by our SDE-EDG, Then, we have
G V2
Ro (h) S (h(
(L(v|p)
(12)
The proof is in E.2 To achieve low risk v, Remark D.2 suggests learning SDE-EDG minimize the KL divergence between & and v, while Re(h) can be approximated by the empirical risk.
In this analysis, we assume that (()) follows ((, T T*) and D(y) is invariant through time In this theorem, analyze the simple case for constant time intervals At between consecutive domains.
Theorem D.3. For any e > O, with probability at least e, the KL--divergence on path space of and & can be upper bounded by:
KL(V|I0) S _ (3 24 + 3M
2
Atlog 2/E
(13)
2TN
where MiH is the Rademacher complexity of the hypothesis set H. M > is the upper bound of & = (Fm 1-8m111f and E is p-Lipschitz for some μ 0. foror any fixed 2m and 2m+1 (2m+1 mapped from 2m).
6(#m Pal
The proof is in Appendix E.6 Theorem D.3 suggests: (1) corresponds to the extrapolation steps on future time. The bound is minimized when L - O. LHS gets larger with a larger L, which indicates the EDG generalizes worse with more future steps. (2) The third term on the RHS gets smaller with smaller At. It implies we should collect sufficient and diverse source domains to ensure that temporal dynamics can learned in the learning process. It also indicates that generating continuous-ing continoous-interpolatee samples will help the learning process. If we collect only one domain even with an infinite number of samples, obviously EDG can not achieved; Equally, we can collect a longer time horizon in the source domains, which means larger T will contribute to better learning evolving patterns and thus a smaller gencralization error. A bigger sample size N in each source domain also contributes to smaller generalization error.
E PROOF OF THEORIES
We first prove intermediate lemma:
Lemma E.1. (Shui et al. 2002)) Let u E === x y be the real-valued integrable random variable, let P and Q be two distributions on common space such that Q is absolutely continuous w.r.t. P. If for any function and X E R such that Ep (eAl) - -Bp((())
< cc, then we have:
(V) Evf(()) S DxL (Q) P) + log Br(eM5tt)-En((to))
where DRu (QI is the Kullback-Leible divergence between distribution Q and P, and the equality arrives when f() = E (v) log( dQ
Proof. We let g any function such that Er E (e)((r) < co, then we define random variable co(e) Vg (v) Ep(enooT then we can verify that Ep(V,) 1. We assume another distribution Q such that
20

---

Published as a conference paper at ICLR 2024
Q (with distribution density (v)) is absolutely continuous w.r.t P (with distribution density p(v)), then we have:
Eqllog V₂l = B& llog g(v) p(v) + log( V
p(e)
= (xl(Q + Eollog(( V p(e)
((0))
'(())
(()) q(v)
- Dxl + log Eol
Vil DKL(Q)))) + log [V]V
Since [V[[V = and according to the definition we have Eollog Vill = Egl((v)] - Eg log Ep Ep [es( = Eols(v)] - log Eples(o) (since Eples(v) is constant w.r.t. Q) and we therefore have:
[alg(v)] s log Eplc"(0) + D
(14)
Since this inequality holds for any function & with finite moment generation function, then we let g(v) ^(f(o) Ep](w)) such that (
xc Therefore we have AA and f we have:
EqA(f(v) - Epf(e)) & (Q] + log Ep(ettttee-)e
Since we have (Q) Ey,((u) = (E)( Ep (())) = - [u)),, therefore have:
(E) - Epf(u)) (KI + log Ep (
As for the attainment in the equality of Eq. (14) we can simply set g(v) log( ( ), then we can
compute Ep(es(v)] 1 and the equality arrives. Therefore Lemma the equality reaches when X(f(v) Erf(v)) log( )
In the classification problem, we define the observation pair e function ((v) = T. h(v) with deterministic hypothesis h and prediction loss function L. Then for abuse of notation, we simply denote the loss function {(v) in this part.
(xr We also define the loss
Then we introduce the following bound between SDE-EDG's stochastic path ú and real evolving path
Remark E.2. (Restatement of Remark D.2 Let á be the learned stochastic path by SDE-EDG, and suppose the loss function is bounded within an interval G : G = max(E) min(e). Then, for any h E H. its target risk R2 (h) can be upper bounded by:
G
(h) S R2(h)+
(L(V))))
V2
where we use KL(-II-) to denote the KL divergence for simplification in the remaining paragraphs.
Proof. According to Lemma
AY > 0 we have:
- Epf(o) < (log Ep (x((0)-+~((1) (QI(P)))
(15)
And AA < 0 we have:
EQf(v) - Epf((e)
(log (logEp Ep ( + Dxx
(16)
Let f= f. Since the random variable is bounded through G = max(f) min(f), then according
we can apply Sub- Gabssias property bound the log moment generation function: 2 then
to (Wainwright 2019) (Chapter 2.1.2), & - Epe is Sab-Gaussian with parameter at most G
log e (((()) < loge 22.2
X²Gº
8
In Eq. 15
we let Q = and P P= D, then AA we have:
G²
Ep ((v) - Ep {(v) < (D'IDD
(17)
8
21

---

Published as a conference paper at ICLR 2024
Since the inequality holds for vX, then by taking X = 2y 6 V DxKI. (D'I)D //xi((D'112)
finally have:
G
Ep {(v) ED {(v)
v (R((DD)
(18)
v2
Let D' and D i0 complete our proof.
Lemma E.3. Define Rya = E 2g as the expected of the model parameterized by 0 with the sample(z, y, f), which is upper bounded by M 0 and is j-I.ipcchizz for some js > 0 for any fixed of and Et. Then, for any >> O, with probability at least
the following inequalities holds
Atg 2/c
Rye S Jpa + 24 3M
(19)
2TN
where Rz is the Rademacher complexity of the hypothesis set H.
Lemma E.3 is directly adapted from Theorem 11.3 in (Mohri et al., 2018).
Assumption E.4. (Realizable) Assume there exists hypothesis ht E H that satisfies
mm++ zm + f.mm fm+t
f (mit) f ge(z,, s)dB.
Ca (=b
(20)
where Ça is the drift function of and gs the diffusion function of h Given same diffusion function shared in Eq. (6) and Eq. (20). the KL between them path space can be approximated (Xu
ct al. 2022 Li et al. 2020).
Lemma E.5. (Restatement of Lemma D.I The optimization loss Jmle defined in Eq. 77 is of the upper bound of the Path Aligument loss Jpa and the Stochasti Uncertainty loss Jew which implies
M K 2
(2
Imic ΣΣΣ
ΣΣΣ
(om) Atm)
2MKNg
(n(em) (2m))AA At
2MKNg
me cm
Jae
Proof. We did not write the superscript the proof for simplicity. Let am 2m+1 generated by the Euler discretization:
Em+1 2m + f(=m Atm g(2m) Bm) 2m + f(=m) Atm g(=m) Atth cm+1
(21)
where (Bm ht.. 20 is the Brownian motion, and 6. is sampled from from 1). This implies that conditional on the previous state, the current state is normally distributed: D(2m+1) 2m) = (2m + (zm Al g(=m) Al.) Thus, the log-densitic can be evaluated as
M
K
N
M
K
N E log D(zm+ilzm)
MKNg Jnnle
log D(= 2m411//2 2/11))
Σ mmm Aem
1ma
22 log(2 gl ) + 1 (2m+1 (Rm + [Em ) Al ))² g(2m)) Ptt
K N
(3m+1 Emt1 + g(zm Atth cm+1) 27 g(2m) Atm
log (zm)))) 2 Atmm))
22
k=l t=1
Ne
1 (2m+1 8m+1)²
+ cm+1 log (2 (Zm ) Atm) + g(2m)22 22m
(2m+1 2m+ 1mm++
(22m) At 1/2
22

---

Published as a conference paper at ICLR 2024
where the second equality is because E +m+ is the feature of real sample and we hide the subscript index i for simplicity.
No
1 Bem+i 1/2
Sm+1
where applies that follows normal distribution.
(2m+1 1m+1)mm+1 to (f(Zm) {(zm)) 2/2 6m+ g(3m) 2/
(22)
g(#m)
where apply the above equation is martingale and its expectation is zero (Xu et al. 2022 Li et al Then, we conclude our proof.
2020
Theorem E.6. The KL-divergence on path space of v and i is upper bounded by:
L
At 2/c
KL(v|10) - 2
Jpa + 2n + 3M
(23)
2TN
Proof.
1. D (2441,8011/2)) - log II 1.
1.
KL(x| E, log
1=1
1=1
( E1, 3141)
D (((+a)
E, log
log
D (zz 21, 8141)
((sc41)
1=1
1=1
D ( (041)
= E,, log
D2 9041)
where in the first line, replace the KL-divergence corresponding the diagram of data generation process and D(t) following U((0, T + 7*) cancels out the Kl--divergence of between D and D; in the second equation, we use the independence of UM++++ #M++ by d-separation (Bishop & Nasrabadi 2006) from figure
Let 2M411 2M42. ZM generated the Euler discretization:
M = 2M+ + ((RM+)) + ((2M)))((M+H)) BM++)
ZM++ + (( At: g(zM+) Ar, EM+++1
(24)
where { h:z is the Brownian motion, and e, is sampled from (0 7). This implics that conditional on the pre- vious state, the current state is normally distributed: 2244141 M((x04 ((*r+t) ((#x++)) Atr). Thus, the log-densities
evaluated
(aMi: (2M44 + At)))
log D(=M+ D(EM+++1l2M4)) = =- (og(279(2x4+)))M
(25)
where:s 1,..... On the other hand, any time step, the in the next domain was generated by observing De, we would have the following log densitii:::
(2M4441 (2M44 + At)))
log D( EM41)
P
(26)
2
g(2ar+))AAl
Now, we replace 2M+++1 in Eq. 24 into Eq. and Eq. (26).
log ((MM4441/2M44) = log (2Tg (=M+)) AAu))
M++++
log(2ng(( (#M++)) Att) (C(za)+)) - f(ZM41)) Al
log ((2M=141/EM=+))
g(2M+1)2
2(C ((((2)+)) f ((ZM))))
st!/2 2204441
g(2M+1)
23

---

Published as a conference paper at ICLR 2024
The KL divergence the path space could then be regarded of infinitely many KL-divergences between Gaussians (applying At, At):
(L(P)P))
//
Ev lim
BEM KL(D(2M+44 htlp:////wlllleeeeeaaaaaaa.a.
Ar-0
1=0
7 /
D(ZM+L+1 SM++: k)
Ep lim
B.g E S
log
50++
D( 238+1, 1:)
1=0
7" //1
(Ca(2 fo fx(2
At + (CA(#Se41) fu(ese-i)) Atf/"entte ea44141 Da (230411)
Ey lim
E E
E B
[ga(#
1=0
=E, E.
lim
/
At
E (ul))
T+T wbdB
2
1=0
//
=E, lim
At
E. (u))²²
A0
2
1=0
where of = (Gi( t) (zt,, f))/ (z+), and E fti,...,fa+ta. In the last cquality, we used the fact that ut dB, is martingale.
the ltô integral J
T /AL
(x(xt) Sa(2)))
Ed( (a(e)) - [a(=t))
E2
Σ
B
2
(x(2))
(x((e)
1=0
(27)
With fixed interval fm+1 fm = +M++++1 tM4+ At, we have
2ētsr = Ca(x,, t) gu(2t (B) Bb)
sttar = =²²² fal t) () B.)
Hence we have
ST
+
B.) B.) fa(e)) gu(s,, (B444, - B:)]l
2gk A2
=L E
2gx (2)) (z))aat At
Hence, we conclude the proof.
F ADDITIONAL EXPERIMENT RESULTS
Evolving Domain Generalization with Continuous Index We evaluate the performance of our method on the continuous index cases by modifying the rotation degrees of the target domains in the RMNIST dataset as (129.1 1", 137.6 144.3", 149.2", 155.5%) We conduct experiments and compare our method with GI, which can also work with continuous index. Compared to GI incremental improvements (0.77 (0.7%) on RMNIST, SDE-EDG could achieve significant improvement with 12.9% higher accuracy on ERM. This shows the superiority of our proposed approach not only in handling discrete index EDG tasks, but also the continuous cases.
Rotated Gaussian Results The prediction results of the last 4 target domains (27-30) are visualized in Figure Rotated Gaussian has a fixed margin p(x) but an evolving p(y(x). ERM has been biased by the source domains, and fails for all target domains by making the opposite labelings. Instead, SDE-EDG could give the correct predictions capturing the temporal patterns (rotating decision boundary by 12" with time).
24

---

Published as a conference paper at ICLR 2024
Table 13: Rotated MNIST (RMNIST) with the Random Continuous Index
ALGORITHM
129.1
137.6
144.3
149.2"
155.5"
AVG
ERM
57.8 = 0.9
45.0
0.9
40.6
1
0.8
40.0
1 0.9
38.3
1
0.8
44.3
GI
59.0
0.8
45.7
0.7
41.8
0.8
39.6
= 0.8
38.9
0.7
45.0
SDE-EDG (OURS)
76.2
0.7
62.9
+ 0.8
54.3
+
0.8
49.6
0.8
42.8
0.8
57.2
Ground
Predictions by
(c) Predictions I.SAF
(b) Predictions by SDE.EDG
Figure 5: (a) The ground truth of the Rotated Gaussian between domains 27-30, and positive and negative labels are red and blue dots separately. (b). (c). (d) show prediction results made by ERM, LSSAE, and SDE-EDG respectively. The dashed line is the ground truth decision boundary of the Rotated Gaussian, which rotates 12a between two consecutive domains. Only SDE-EDG could effectively capture the exact evolving patterns, achieving the best performance on Rotated Gaussian.
G ADDITIONAL ABLATIONS
G.1 IMPACT OF DIFFERENT TIME INTERVALS ON BASELINES
The experiments applying various intervals for the baselines show:
Smaller time intervals generally enhance the performance all methods.
With different intervals, our method, SDE-EDG, consistently outperforms the other base- lines.
Table 14: RMNIST with time interval as At/2
INTERVAL= 212
130
140"
150"
160"
170"
180
AVG
ERM LSSAE
60.7 + 0.9
46.6
0.8
39.4
0.8
0.8
39.8 + 0.6
41.0
0.8
44.3
65.5 0.8
52.6
45.2
3..0
40.1
41.1
0.9
47.3
SDE-EDG (OURS)
75.6 = 0.8
61.8
49.9
50.0 0.9
45.1 = 0.7
44.1
0.9
54.4
Table 15: RMNIST with time interval as 2At
INTERVAL= 2A
130"
140"
150"
160"
170"
180
AVG
ERM
47.2 0.8
37.9
32.9 0.8
34.9
38.1 1.10110
1.0
43.8
0.7
39.1
LSSAE
49.3 0.9
38.8
34.9
3.9
40.6
0.8
40.7
0.8
40.4
SDE EDG
58.6 1 0.8
10.7
45.6 0.7
42.4 0.8
36.9
0.8
36.1
0.8
44.8
G.2 ABLATION OF E (0.8, 1,3,5,,7)79)
In the ablation study, the hyperparameters were varied within a wide range (0, 0,1, 1, 10, 100, 200). causing notable fluctuations in the results. On the other hand, within the range of [0.8, 10), the model's performance remains relatively steady (in Figure 4 (d,e)). The subsequent ablation experiments further show the stable performance by setting
{0.8, 1.
In practice, we select hyperparameter a based on the validation set. Without employing cross- validation, still can set 1 yields consistent results shown in Figure 4 (d,e).
25

---

Published as a conference paper at ICLR 2024
Table 16: RMNIST with [0.8, 10]
130"
140"
150"
160
170'
180
AVG
72.7 +
59.8
0.7
49.6 0.8
49.2
0.7
+
39.5 0.6
51.5
75.1 +
61.3
0.8
0.9
39.7
0.7
39.7
52.6
1
58.8
4.7
0.7
40.7 0.8
39.3
51.2
73.6
58.1
4.9
4.8
39.0
39.7
50.2
76.7 1 0.7
6..1
50.5
0.8
4.9
39.0
0.8
09.1
51.7
73.7 +
59.8 0.7
47.9
39.4 4..0
0.6
50.6
G.3 ABLATION ON UNI- OR MULTI-MODAL CLASSIFICATION LOSS
We conduct the experiments with uni-modal and results reveal that the performance of the uni-modal approach closely resembled that of the multi- modal. The choice between the modalities often hinges upon the inherent characteristies of the datasets themselves. In real-world applications, volssvvaiiation can be applied to determine the most suitable method for a specific dataset.
classification loss respectively. The
Table 17: Ablation on uni- and multi molli-modal loss
DATASESS RMNIST
PORTRAIT POWERSUPPLY
UNI
52.4
89.6
75.7
MULTI
52.6
88.9
75.1
26


Reviews:
[{"review_id": "2VkEFQ2PZ4", "reviewer": "Reviewer_bSwA", "summary": "The present work proposes SDE-EDG, a novel learning approach for solving the problem of Evolving Domain Generalization in the context of (neural) Stochastic Differential Equations. Real-world applications pose challenges to existing EDG approaches, such as inconsistent data availability over time, which leads to difficulties in capturing the underlying dynamics, risks overfitting and reduces the quality of generalization to unseen data. Further, distributions that change over time cannot be taken into account as they contradict the stationarity assumption. The authors claim to overcome all those limitations by incorporating the learning of latent trajectories that describe the distributional changes over time. The latter is done using variational inference in the setting of neural SDE approaches and by improving latent trajectory fits by bridging temporal gaps through linear interpolation of samples from consecutive distributions (the IFGET module). Empirical evidence is provided through synthetical and real world experiments demonstrating superior performance compared to existing state-of-the-art methods.", "questions": "1. (p.3) Do you assume all source domains have the same sample size $N$ or can it vary?\n2. (p.4) You repeat here (unnecessarily, in my opinion) the argument of \"sample complexity\", which already appeared on p. 1. However, I would like to ask if you could explain this argument in more detail?\n3. (p.4) Can you define $N_B$?\n4. (p.4) Why do you choose to draw the interpolation rate $\\lambda$ from a Beta distribution?\n5. How many interpolation points are generated between two consecutive data points, i.e., how many $\\lambda 's$ are drawn; am I right in assuming that only one is drawn? Would an increase in the number of intermediate points result in better performance?\n6. Neural SDEs are known to be very resource consuming. Can you assess the runtime complexity, for example?\n7. (Figure 4(c)) Can you assess what happens at Domain Index 29; why does SDE-EDG show a reduction in performance?\n\nMinor:\n    - Redundancy in the references.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The generative approach is not only convincing due to its quantitative superiority, but also provides insights into the black-box mechanisms of model learning, which can be easily generalized to unknown target areas. The authors exemplify how such investigation of representation in the form of latent trajectories (e.g., see Figure 2 and 3(d/e)) provides valuable insights. Further, empirical evidence is accessed through experiments on synthetical and real-world data as well as reproducibility is granted by submitted code files. The paper content is original to the best of my knowledge and, with few exceptions, has no spelling or grammatical flaws. Finally, on a positive note, the authors perform a series of ablation studies to test the validity of their claim that the proposed IFGET module contributes complementary information to learning evolving dynamics in a classical neural SDE scheme.", "weaknesses": "1. Learning neural SDEs is traditionally successfully presented in the framework of variational Bayesian inference (e.g. Li et al. 2020), where stochastic gradient descent methods are applied to minimize the evidence lower bound (ELBO). I wonder why the authors do not clearly address the reference to variational Inference in the manuscript, especially since the indications (e.g., the graphical model in Figure 1, derivation of the likelihood loss of IFGET in Lemma B.1) are given.\n\n2. The Sine experiment (Figure 3) is arguably a rather simple problem from a dynamic point of view and yet already indicates the limited extrapolation quality of the approach. Of course, this reduces my confidence in the generalization ability of the approach. I suspect possible causes in (i) an unfavorable modeling of drift and diffusion coefficients and (ii) that the linear interpolation approach in the IFGET module is too coarse. Can you comment on this?\n\n3. The IGET module reminds me on a approach by Kidger et al. 2020 (Neural Controlled Differential Equations for Irregular Time Series) in which the authors also try to improve the learning of latent trajectories by including interpolated sample trajectories, in their terms a \"controlled path”. This is done in the ODE setting, but can be integrated into the SDE setting if the approach is applied to learning the drift coefficient. Can you explain in more detail how your approach differs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rikiyzpvfP", "reviewer": "Reviewer_KuA7", "summary": "This paper investigates the evolving domain generalization (EDG) task. Previous works have difficulties in capturing evolving dynamics due to limited timestamps. To address this, this paper simulates the data distribution with continuous-interpolated samples and leverages SDEs to capture evolving distribution trends. The proposed method achieves SOTA performance on all 9 benchmarks.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method achieves state-of-the-art performance on the benchmarks.\n\n2. Extensive experiments and analysis demonstrate the effectiveness of the proposed method.", "weaknesses": "1. In one iteration, for two consecutive domains, only one interpolated sample is generated. Can more than one sample be generated and used? For example, we can first generate one interpolated sample via Eq.5 and then set this sample as $\\tilde{z}_{m+1}$ to generate the second sample. If so, how does this influence the performance?\n\n2. Table 2 shows that smaller temporal gaps improve the generalization ability. How is the improvement of SDE-EDG over the baseline when using different time intervals?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qnFldN5UDK", "reviewer": "Reviewer_y1vq", "summary": "This paper proposes a SDE-EDG method in order to solve the challenges that limited number of timestamps are available in evolving domain generalization problem via collecting infinite fine-grid evolving trajectory of the data distribution with continuous-interpolated samples to bridge temporal gaps.", "questions": "1. What are the differences between continuous-interpolated samples and samples used in previous works?\n\n2. From my perspective, the interpolated samples are generated discretely. Thus, does the SDE-EDG method approximates the continuous case in the way of discretely sampling?\n\n3. According to my understanding, Eq.(7) aims at learn a set of parameters for $f_k$ and $g_k$ so that the learned models can mimic the approximated trajectory of data. In my opinion, such operation relies heavily on the quality of data representations, especicially at the early phase of training, since the feature extractor $\\phi$ is not well trained yet. Thus, in such case, will the linear interpolation results in some \"bad interpolation\"?\n\n4. According to the ablation study, the hyper-parameter $\\alpha$ seems sensitive for different datasets. Since $\\alpha$ scale the magnitude of $\\mathcal{L}_{mle}$ that plays the role of regularizer, could you explain such phenomenon?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The writting of the paper is good.\n- The problem studied in this paper is well and clearly formulated.\n- The illustration and demonstration is convincing and solid.\n- The proposed SDE-EDG achieves better empirical results on several benchmark datasets than existing SOTA methods.", "weaknesses": "- The illustration of continuous-interpolated samples is not very clear. I am a little bit confused about why the samples generated with linear interpolation method are called \"continuous\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DtBOvxBVrD", "reviewer": "Reviewer_pVBo", "summary": "Considering the evolving characteristics of data distribution is crucial for machine learning models in practical applications. Such problem is formalized as Evolving Domain Generalization (EDG) in literatures. This paper a common issue in EDG where limited timestamps can lead to overfitting to source domains. In tackling this challenge, this paper introduces a novel approach that involves gathering the Infinitely Fined-Grid Evolving Trajectory (IFGET) to capture evolving dynamics and align stochastic distribution of Stochastic Differential Equations (SDEs) with IFGET. This alignment effectively captures distribution shifts across the sequence, enabling SDE-EDG to adapt the model for generalization in dynamic environments. The experimental results, conducted on various synthetic and real-world datasets, provide empirical evidence of the method's effectiveness. Overall, I believe this work proposes an innovative solution to the challenging yet under-studied problem of EDG, representing a valuable contribution to the field.", "questions": "The computational costs of Neural SDEs, particularly regarding backward gradient propagation, are relatively high. However, this paper does not provide discussions of this aspect.\n\ntypos: in section 5.1, two repetitive \"Ocular Disease\"; \n\nIn section 4.4, you should not use the same $k$ in the summation of denominator for $\\frac{ |\\mathbb{S}^k_m|}{\\sum_{k=1}^K|\\mathbb{S}^{k}_m|}$", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1 Infinitely Fined-Grid Evolving Trajectory (IFGET) method is novel. Capturing evolving dynamics but avoiding overfitting to limited source timestamps is achieved by collecting IFGET in the latent space using continuous interpolated samples, a strategy that has demonstrated its effectiveness in ablation experiments. Specifically, this paper proposes to collect IFGET such that the evolving patterns are learned from trajectories of the individual sample instead of collective behaviors in existing EDG methods. Therefore, IFGET provides more accurate evolving trajectories by tracking individual sample's behavior, providing a finer-grained understanding of individual sample behavior.\n\n2 To the best of my knowledge, this is the first work to introduce Stochastic Differential Equations (SDEs) to address EDG tasks. The utilization of SDEs for modeling the continuously evolving trajectories of the latent space in the EDG problem is natural because of the inherent capabilities of SDEs in characterizing continuous stochastic processes.\n\n3 Figures 2 and 3(3) serve as clear illustrations of the superior performance achieved by capturing evolving dynamics with IFGET, affirming its indispensable role in addressing the challenges within the EDG problem.\n\n4 This paper is well-presented and easy to follow. The paper demonstrates the effectiveness of SDE-EDG on various datasets, including simple and complex datasets, showcasing its outperformance compared to other baseline methods by effectively capturing evolving dynamics.", "weaknesses": "1 I doubt the existence of evolving dynamics. It's worth considering scenarios where time-related tasks might lack evolving patterns and instead exhibit random shifts.\n\n2 Linear interpolations may not reflect the true evolving trajectories, since the real evolving trajectories might be nonlinear and complex.\n\n3 Considering that IFGET is an approximation, and the existence of ground truth sample-to-sample correspondence is uncertain, how do we guarantee that IFGET accurately represents the real evolving trajectories?\n\n4 The authors conducted an ablation on weighting on IFGET; however, I am curious about the performance of IFGET without continuous interpolations, which will show whether continuous interpolations improve generalization to sparse timestamps.\n\n5  I understand neural SDEs that neural SDEs do not rely on the assumption of the model being uni-modal, in contrast to prevelant uni-modal Gaussian distributions. However, most deep learning methods assume the classification tasks to exhibit uni-modal characteristics through using ERM classification loss. Therefore, I am wondering whether uni-modal or multi-modal classification loss contributes to the accuracy improvement of SDE-EDG in EDG.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)