Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
Authors: Zhuo He
Shuang Li
Wenze Song
Longhui Yuan
Jian Liang
Han Li
Kun Gai
Venue: ICML 2025 poster (2025)
Keywords: Evolving Domain Generalization
Causal Representation Learning
Primary area: general_machine_learning->transfer_multitask_and_metalearning

Abstract: Endowing deep models with the ability to generalize in dynamic scenarios is of vital significance for real-world deployment, given the continuous and complex changes in data distribution. Recently, evolving domain generalization (EDG) has emerged to address distribution shifts over time, aiming to capture evolving patterns for improved model generalization. However, existing EDG methods may suffer from spurious correlations by modeling only the dependence between data and targets across domains, creating a shortcut between task-irrelevant factors and the target, which hinders generalization. To this end, we design a time-aware structural causal model (SCM) that incorporates dynamic causal factors and the causal mechanism drifts, and propose **S**tatic-D**YN**amic **C**ausal Representation Learning (**SYNC**), an approach that effectively learns time-aware causal representations.  Specifically, it integrates specially designed information-theoretic objectives into a sequential VAE framework which captures evolving patterns, and produces the desired representations by preserving intra-class compactness of causal factors both across and within domains. Moreover, we theoretically show that our method can yield the optimal causal predictor for each time domain. Results on both synthetic and real-world datasets exhibit that SYNC can achieve superior temporal generalization performance.

Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
Zhuo He'
Shuang Li
Wenze Song'
Longhui Yuan
Jian Liang?
Han Li
Kun Gai
Abstract
Category Distribution
160
140
Endowing deep models with the ability to gener- alize in dynamic scenarios of vital significance for real-world deployment, given the continuous and complex changes in data distribution. Re- cently, evolving domain generalization (EDG) has emerged to address distribution shifts over time. aiming to capture evolving patterns for improved model generalization. However, existing EDG methods may suffer from spurious correlations by modeling only the dependence between data and targets across domains, creating shortcut be- tween task-irrelevane factors and the target, which hinders generalization. To this end, we design a time-aware structural causal model (SCM) that in- corporates dynamic causal factors and the causal mechanism drifts, and propose Sratic-DNamne Causal Representation Learning (SYNC), an ap- proach that effectively learns time-aware causal representations. Specifically, integrates spe- cially designed informatiion-hhoorttii objectives into sequential VAE framework which captures evolving patterns, and produces the desired repre- sentations by preserving intra-class compactness of causal factors both across and within domains. Moreover, theoretically show that method can yield the optimal causal predictor for each time domain. Results on both synthetic and real- world datasets exhibit that SYNC can achieve superior temporal generalization performance. ?
No Car
Creoning
Wooe
Diay
Eventing
Dany
Eveniin
Regitd
Dity
Dlay
Expoming
Attention Map
Evening
330
Night
Day
Evening
True label
No Car
No Car
Car
LSSAE
Prediction
No Car
No Car
Figure 1:
We plot the category distribution training do- mains on Caltran. The lower part shows visualization of attention maps of the convolutional Layer. Compared with ISSAE, be observed that method better focuses target-related features and makes accurate predictions.
eralization (DG) (Liu al., 2021b; Wang et al., 2023; Fan et al., 2024) can effectively address the issue of distribution shift, but it still struggles to handle the dynamic scenarios that are widespread the real world (Yao et al., 2022a; Qin et al., 2022). order to adapt to changing environments over time, evolving domain generalization (EDG) (Nasery et al., 2021; Qin et al., 2022: Zeng al., 2023b; Bai et al., 2023; Xie et al., 2024; Zeng al., 2023a; Qin al., 2023) has emerged in recent years and is garnering increasing at- tention, aiming to capture the underlying evolving patterns in data distribution to generalize well to the near future.
1. Introduction
Deep neural networks (DNNs) have achieved excellent performance in various applications, yet suffer from per- formance degradation when the i.i.d. assumption is vio- lated (Recht al., 2019; Taori al., 2020). Domain Gen-
Despite promising results, existing EDG methods may suffer from spurious correlations by solely modeling the statistical correlation between data and targets. Fig. illustrates the behavior of the baseline model (Qin et al., 2022) trained on the Caltran dataset (Hoffman et al., 201 4), tasked with deter- mining whether a vehicle is present in an image. Among the images captured by the camera, most those containing vehicles were taken during the day, while images without vehicles were captured at evening or night. Therefore, there
'Independent Researcher, China Kuaishou Technology, China. Correspondence to: Shuang Li <shuangliai @ buaa. edu.eed..cn......
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
and dynamic factors Zdy, respectively. Therefore, in our new SCM, the data X is generated by X <- (Z Zdu) while the label Y is generated via Y < (Z _ 2d). It can be observed that the time factor acts as a latent con- founder, with its components G and L forming backdoor paths between static and dynamic spurious factors and the target. These paths introduce spurious correlations, which can be mitigated by identifying the true causal factors and constructing causal model accordingly.
E
Z
G
s
c
X
Za
zdy
zay
L.
X
Y
In order to learn the causal model with the help of the time- aware SCM, we propose Static-DYNanie Causal Represen- tation Learning (SYNC), a method designed to effectively capture tame- causal representations by simultaneously uncovering static and dynamic causal representations. Con- cretely, sequential VAE framework is employed to capture the underlying evolving patterns in data distribution. To sep- arate causal factors from the complex and entangled mixture of factors, we first minimize the mutual information (MI) loss between static and dynamic factors to encourage dis- entanglement between them. After that, static and dynamic causal factors can be obtained by maximally preserving the intra-class compactness of causal factors both across and within temporal domains, implemented by optimizing the designed cross- moos--domain and intra-domain conditional MI losses that account for continuous domain structure in EDG, We theoretically show that, with the devised objectives, our method can yield the optimal causal predictor for cach time domain, ultimately excluding spurious correlations and achieving superior temporal generalization performance.
(a)
(b)
Figure 2: (a) SCM for DG. (b) Our proposed time-aware SCM for EDG. By introducing z and 24, dynamic causal factors and causal mechanism drifts are explicitly modeled.
exists a shortcut from the background representation to the target, i..., the trained model tends to rely on lighting (spu- rious feature) to determine the presence of vehicle, rather than the object of the vehicle (causal feature). Due to spuri- ous correlations, the model during the inference stage tends to classify day (night) images as "Car" ("No Car"), which compromises its generalization performance.
Causality (Pearl, 2009; Pearl et al., 2016) has proven to be a powerful tool for addressing spurious correlations and has therefore been widely explored. (Arjovsky et al., 2019; Mahajan al., 2021; Sun al., 2021; Lv al., 2022: Sheth et al., 2022). However, since causal factors and causal mechanisms behave more complex and changeable in non- stationary environments, its implementation in dynamic scenarios still remains underexplored. Fig. 2a shows a clas- sical structural causal model (SCM) for static DG (Liu et al., 2021a; Sun et al., 2021; Lv et al., 2022), where category- irrelevant (spurious) factors Z, and category-related (causal) factors Z. modeled as inne-invaiiant. Such a temporal homogeneous SCM fails to account for the underlying con- tinuous structure in non-stationary tasks, rendering static causal representations insufficient for generalizing to evolv- ing new domains. Moreover, as the statistical properties of the target variables change over time (Gama et al., 2014: Lu al., 2018), the inffuence of causal factors on the target may also vary, suggesting the mapping from causal factors to labels gradually shifts, leading to causal mechanism drifts.
Contributions: (i) By taking novel causal perspective towards EDG problem, we design a time-aware SCM that enables the refined modeling of both dynamic causal factors and causal mechanism drifts. After that, we propose SYNC, an approach for effectively learning time-aware causal rep- resentations, thereby mitigating spurious correlations. (ii) Theoretically, we show that SYNC can build the optimal causal predictor for each time domain, resulting in improved model generalization. (iii) Results on both synthetic and wolld datasets, along with extensive analytic experi- ments demonstrate the effectiveness of proposed approach.
2. Related Work
To address the aforementioned issues, we propose novel time-aware SCM for EDG. As shown in Fig. 2b, the local variable L introduced to model instability and dynamics, and the global variable G is utilized to represent globally unchanging knowledge. We then split the causal factors into static causal factors Z and time- vaariig dynamic causal factors Zdy, thereby incorporating the evolving patterns into causal factors. To model the causal mechanism drifts, we introduce a time-varying drift factor Zd. Moreover, Z and Zdy, along with the static spurious factors Z and dy- namic spurious factors zdy constitute the static factors z
Evolving Domain Generalization (EDG) is introduced to address the issue of generalizing across temporally drift domains (Nasery et al., 2021; Qin et al., 2022; Zeng et al., 2023b; Bai et al., 2023; Xie et al., 2024; Zeng et al., 2024; 2023a; Qin et al., 2023; Jin et al., 2024). Most existing methods learn a time-sensitive model. To name a few, GI (Nasery et al., 2021) suggests training a time-sensitive model by utilizing gradient interpolation loss. LSSAE and MMD-LSAF (Qin al., 2022; 2023) account for covari- ate shifts and concept drifts and introduce probabilistic
2

---

Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
model with variational inference to capture evolving pat- terns. MISTS (Xie et al., 2024) further proposes to learn both invariant and dynamic features through mutual infor- mation regularization. Additionally, DRAIN (Bai et al., 2023) launches a Bayesian framework and adaptively gen- erates network parameters through dynamic graphs. SDE- EDG (Zeng et al., 2023a) proposes to learn continuous tra- jectories captured by the stochastic differential equations. Different from the aforementioned methods, DDA (Zeng et al., 2023b) simulates the unseen target data by meta- learned domain transformer. However, these methods do not account for spurious correlations, which could hinder the model generalization. In this work, we introduce a novel causal perspective to EDG problem and learn time-aware causal representations to address this issue.
given T sequential source domains S = (Dilf= where each domain Dj consists of Ne data (dzr,s, Ut,i) which drawn from P(x, y)t,, i.e., Dr = {( goal of EDG is to leverage given T training domains to
) ) The
establish a robust model h X + y which can generalize well to K unseen target domains T (D:) arriving
sequentially in the near future. Due to the existence of evolving patterns across the sequential domains, the ideal model should be capable of capturing these patterns.
Structural Causal Model. The structual causal model (SCM) over random variables X1, X2,... .,,, accord- ing to (Pearl, 2009), is defined as triple P(E), F). G denotes the causal directed acyclic graph (DAG) which is comprised of the nodes V and the directed edges E, where the starting point and end point of each directed edge rep- resent cause and effect, respectively. E is the independent exogenous noise variables. These noise variables follow the joint distribution P(e). F denotes set of assignments {X, := fa (PA, ==) where PA, is the set of parents of X,, iee, direct causes, fi represents a deterministic function. Each SCM induces a corresponding causal DAG, and the data generation process can be characterized by the SCM.
Causality and Generalization. Causality describes the fundamental relationship between cause and effect, which goes beyond the statistical correlation simply obtained from observational data (Pearl, 2009; Pearl et al., 2016). Since causal models can effectively mitigate the impact of spu- rious correlations, many studies have proposed leveraging causal theories to develop more robust models. Causal meth- ods in DG can be broadly categorized into two types. The first type of methods (Arjovsky et al., 2019; Ahuja et al., 2020; Li et al., 2022; Ahuja et al., 2021; Sun et al., 2021; Yong et al., 2024) focuses on constructing an optimal in- variant predictor through invariant learning. However, by neglecting dynamic causal information or the correlations between temporal domains, these methods struggle to per- form effectively in dynamic environments. The second type (Mahajan et al., 2021; Lv et al., 2022; Wang et al., 2022; Hu et al., 2022; Mao et al., 2022; Chen et al., 2023; Yue et al., 2021) leverages interventions to extract causal features. Nevertheless, in dynamic scenarios, the presence of dynamical causal factors complicates the rational inter- vention in spurious factors. Regarding causal methods for time series modeling (Entner Hoyer, 2010; Malinsky & Spirtes, 2018: 8: Li et al., 2021; Yao et al., 2022b), they often need somewhat strong assumptions on models such as the reversibility of the generation function to infer underlying causal structures. Furthermore, the aforementioned methods overlook the fact that the causal mechanisms of the target may drift (Lu et al., 2018) in dynamic scenarios.
3.2. Temporally Evolving SCM
Spurious correlations may mislead the model into learning features that are irrelevant to the target, posing a notable challenge to model generalization in EDG. Since causal models can effectively alleviate spurious correlations, we aim to construct an appropriate SCM for EDG and acquire the causal model by learning causal representations.
Given that muusal-based methods (Lv et al., 2022; Liu et al., 2021a; Lu et al., 2021; Sun et al., 2021) have been exten- sively explored in static DG, we start by briefly reviewing the SCM employed in static DG. As mentioned in Sec- tion 1, the target Y is determined solely by causal factors Ze. and the causal mechanism Z(Y remains consistent across environments. On this basis, causal model can be established by extracting Ze and constructing the corre- sponding invariant causal mechanisms. Nevertheless, such time-independent SCM is not suitable for dynamic scenar- ios. Specifically, first, ignoring the underlying continuous structure stationtt tasks limits performance. Sec- ond, the statistical properties of the target may change over time (Gama et al., 2014; Lu et al., 2018), implying that the influence of causal factors on the target may vary, resulting in causal mechanism drifts. Therefore, to apply causality in EDG, designing a new and suitable SCM is essential.
3. Methodology
3.1. Preliminaries
Evolving Domain Generalization. Let E X' and y E y denote the input data and its label, respectively, where X and y' represent the nonempty input space and label space, respectively. P(æ, y, I) characterizes the temporal dynamic of the probability distribution for (a, y), wherein underlying evolving patterns over time exist. Suppose that we are
Regarding the first issue, we further partition the causal factors into static causal factors Z and dynamic causal factors Zdd where Z contains stable category-related in- formation, while Z is related to domain-specific category- related information. z and Zdy, along with the static
3

---

Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
spurious factors z and dynamic spurious factors Z.4. constitute the static factors 7 and dynamic factor 7dy, respectively. Then, Zot and Zdy collaborate to generate the observed data X. Regarding the second issue, in order to characterize the evolving causal structure P( " we introduce drift factors z to characterize the evolution of causal mechanisms. Finally, we introduce the local vari- able L. which consist of unstable and dynamic information, along with the stable global variable G, represent the generation of zo, Zdy and 2á. Combined with the above analysis, we propose the time-aware SCM as follows:
2
Rt-
Z
by
aay
2
zdy
Figure 3: Temporally Evolving SCM for EDG. The dashed line indicates that the two factors are correlated.
Definition 1 (Time-aware SCM). The time-aware SCM MT built on observed variables X, YY, latent variables Ze Z Z - ; zd and information factors G and L, is a triple, i.e., MM = (G. F()) 9 is the corresponding directed acyclic causal graph shown in Fig. 2b, P(c) de- notes the distribution that exogenous noise follows, and F denotes the assignments { """.f".e fat fst fdy
rational intervention in spurious factors; (ii) Learning with invariant causal mechanisms, as in a static environment, is not feasible. To address these issues, we propose Static- DYNamic Causal Representation Learning (SYNC), which effectively learns time-aware causal representations com- posed of both static and dynamic causal components.
1e.5e f fr::
= (Ze', 2,,, 2eu,25 z.a, y ; (Ze ,2 oez ,,),,
3.3. Static Causal Representation Learning
2.¹h = (G, ,, 1;; (G,,),
- = (L, Co dr) -.5 :- 1.
d zz :- (L,
3. EVOLVING PATTERN LEARNING
We consider the evolving patterns of both static and dynamic latent factors. For data (æx, Ve) collected at time stamp t, let 2 and x denote the static factors and dynamic factors, respectively. Our goal is to learn the latent evolving patterns by modeling the condition distributions p(2;" (æ:;) and (( dy
From Def. I, the generation of the observed variables fully determined by five latent factors, with no additional unob- served factors involved. Under the global Markov condition and the causal faithfulness (Pearl et al., 2016), it follows that Y IL [z Zdd) (2et, Zág 24). This observation suggests that, within a given time domain Dr, predictors built on causal representations are immune to the influence of spurious factors. By incorporating a mutual information (MI) term to filter out information irrelevant to the target, the optimal causal predictor for D, can be defined as follows:
<<t xt). To this end, a neural network qo (=(z) du dg xt) = -N(u(z("). (z(")) is employed to ap- dy dy proximate p(2, =<ct mn). This approximation is achieved dy | x1), p(z, | xx))) 1=
by minimizing BKi(g0 (z (90(2 which is equivalent to maximizing
Eg. log p(xxp/(/()
Dx (1o
220 dy 2x), pl p(z
Definition 2 (Optimal Causal Predictor). For a time domain Dr, the corresponding drift factors Z are given, Let Ze,t= (Zet, zdy) )
is the prior distribution of x Inspired by (Qin et al., 2022), we model it using LSTM (Hochre- iter & Schmidhuber, 1997), by setting the initial state as = 0. For p(2;") (at). we utilize a network go (2f"|z;) = Nu o² (z+"")) ) to approximate. Analogously, to mini- mize DxI(ge (go (z," p(=)) (a.)). we need to maximize
where p( -( |
be the causal factors in domain Dr that satisfy Ze,t E arg max Z2.1 I(Y; Ze,t, 2 and Y IL [Zet 2 (Z,, 7 Then the predictor based on factors Ze,t and z is the optimal causal predictor.
From Def. 2, when static and dynamic causal factors, along with drifts in causal mechanisms, are modeled together, the causal model for each time domain can be accurately learned. Building on the time-aware SCM My, we formu- late a temporally evolving SCM Mevo, which is composed of Mr as the basic element, connected sequentially along the time axis. As shown in Fig. 3, static factors remain invariant to temporal changes, whereas dynamic and drift factors evolve according to specific patterns.
Eg log p(xx//1)))) 11 1k (2x p()))(=?)))
where p(z;") is set as /O, I). In general, the objective function for learning the latent patterns of features is
Lip Σ
Eggn Dog 2d0)]
+
(I)
Existing causal methods developed for static DG suffer from the following two problems in non-stationary environments: (i) The presence of dynamic causal factors complicates the
DEr(go dy 2 xx) p(z dy 1
+

---

Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
Mir. dy
Evolving Pattern Learning
Causal Representation Learning
la:ee a a
posterior
FET
Static Causalizer
4
sample
Static Causalizer
D
X1
oo
EMD
qu
sample
Dynamic Causalizer
6.1.7
w
Dynamic Causalizer
dAp
at oe
of
Static (Dynamic) Causalizer
m
(m
dy
YuT a
sample
1
Levalve
prior
qo, Figure 4: The training framework of SYNC. We employ Q0 9 to model the posteriors, where oj = (Hii of"), n, o²
the mean and variance of the posterior, denotes "st", "dy" or "d". Then, the sampled zitr and =1.7 are utilized to reconstruct the data through the decoder D. After passing the corresponding causalizer (in which m or m is 0-1 masker), they are combined with <1.r to make predition through the classifier The reconstruction term and the divergency between the priors and the posteriors is used to calculate the evolving pattern loss Levolve. The means of the static and dynamic posteriors are utilized to calculate MI losses LMI, faxe, Laye, aiming to obtain static and dynamic causal factors.
where the log-likelihood term represents the reconstruction term for input Tt, two KL divergence terms are used to align the posterior distributions of 2 and zi with the corresponding prior distributions.
3.3.3. UNEARTHING STATIC AND DYNAMIC CAUSAL REPRESENTATIONS
As mentioned in Section 3.2, learning causal representations poses significant challenges. To address this, we need to find some properties of causal factors and translate them into equivalent and feasible objectives. Prop. 1 below gives the properties of causal factors in the continuous time domain:
3.3.2. DISENTANGLEMEEN OF STATIC AND DYNAMIC REPRESENTATIONS
Although static and dynamic factors are modeled sepa- rately in Eq. (1), the absence of additional constraints pre- vents us from guaranteeing that static factors do not con- tain dynamic information, the same is true for dynamic factors. MI quantifies the degree of dependence between two variables. By minimizing the MI, the exclusivity of the information contained in these variables can be pro- moted. Therefore, in order to achieve the disentanglement of zz' and dy we propose to minimize the MI between them. The MI between 2 and 2, can be written as T(27"; = (22")) + H²2a)) H(2;", > Follow- ing (Chen et al., 2018b), we use standard mini-batch weighted sampling (MWS) to estimate the entropy term (-) = [log [log (()).. Specifically, when provided with a mini-batch of samples {o/, xf, ... xx) at time t, we can use the estimator
Proposition 1. Let Dj be time domain, and for given class Y. it can be conclude that:
(i) If there is H(Zst 7st then 1(Zzt; 2et Y) ((2g. 221-111).
c,t -11-11 Y) < H(Zat) Zet [
c,t-1 Y),
(i) If there is H( Zuy Y) HH Y). then 1(Z2) 2 > 1 I(Zdy YY
dy
In fact, the condition in Prop. is generally easy to meet, as causal factors typically exhibit greater similarity, leading to smaller conditional entropy.
Remark 1. Although Prop. I appears straightffrward, its in- spiration comes from My and can be intuitively understood within this framework. Since the two claims are similar, we explain only the second one for simplicity. As shown in Fig. 2b, there exists a collider Z + Y + Zudy and a L + Zady According to d-separation (Pearl
B
B BB
g(:)
=0 B
log
Σ {(<<<)))))1 1=1
(2)
fork 2 et al., 2016), when conditioned on Y, both 2 and 2 are related to Z Consider boundary case where Z and 7:4y are independent, this implies that the backdoor path between Zdy and Zdy is blocked, Icading to independence between Znd and Zddy while Z remains related to Z
where =xx) denotes a sample from d(-1=)), and B and B' denote the batch size and data size, respectively. The MI loss can be written as
T LMI
1( zdy)
(3) Prop. generalizes this observation, under certain entropy inequalities, static causal factors are more strongly related
1=1
5

---

Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
dynamic causal factors than dynamic spurious factors. As mentioned above, the conditions are easy satisfy.
such that Φ m o gg mg is implemented is similar to Eq. (5). We estimate T( X)); Z (V) by
Prop. 1 (i) provides a way to learn the static causal factors:
e G//)/
E (f (( log e(("1 + E CF
max I( (X.); D2 (Xx-1)) t1
(4)
(xg)) (PXX)
where 1 (j. si
2- (XP,x
where Th consists of the feature extraction component of Lú. namely g and neural-network- masker me
sim ( (Xt,), 2. (XP.).. with X2, being a positive sam- ple that shares the same label as Xe.j. and the negative sam- ples M { are those having different labels. 2%x² (X;,)
such that कet m qØ m²² is implemented by
denotes the static causal factors of Xi.. which are imple- mented by (X).). We provide the following Lem. 2, which demonstrates the effectiveness of devised objective.
st = Gumbel-Softmax (g cat (Xe)))))))))) (X+)), kN),
(5)
where w learns the scores of each dimensions of q (Xt),
Lemma 2. For a time domain Dr, let Φzt.* (X,) be the static causal factors learned by the model, then the dynamic causal factors can be obtained by the following objective:
and the dimensions correspond to the mask ratio i E (0.
are regarded as causal dimensions, N is the dimension of eext (X.) We employ the Gumbel-Softmax trick (Jang et al.,
2017) to sample a mask with xNV values approaching 1.
max I(Y; (X.)),
Since an exact estimate of Eq. (4) could be highly expen- sive (Oord et al., 2018), we use supervised contrastive learn ing (Khosla et al., 2020; Belghazi et al., 2018) as a practical solution for the approximating 7(055 (Xt); x-t (X.-)))))
(10)
S.1. Tayy E Earg max T( do (Xt); t5t (X.IV))
Overall, the objective in Eq. (4) brings the static causal repre- sentations of the same category across domain closer, while the objective in Eq. (8) aligns the static and dynamic causal representations of the same category within the domain.
ol U'NN
(x aa ( log Jf +EM e''GG 1x (
(6)
where (i,k) sim( (Xis), D (Xp-_,)) ) and (. sim sim( (Xe,)); &." (XT sim(..) the cosine similarity. In Eq. (6), the positive sample X²_ shares the same label as Xt.j. while the negative sam- ples 1 M have different labels. 7 is the tempera- ture hyperparameter. In addition, we utilize the MI term T(Y; ".": (X+)) to filter out information irrelevant to the tar- get. Lem. as follows demonstrates the effectiveness of designed objective.
3.3.4. LEARNING EVOLVING CAUSAL MECHANISMS
We introduce drift factors 2d to model the evolving causal mechanisms. In order approximate p(= < Vt), we use a network parameterized by recurrent neural network qc (=fl vt) with a categorical distribution as the output. The initial state is set as 2 0. Then minimizing the KL divergency between q and p can be achieved by maximizing
Egg log p(ppl1l (x Bc), pl 2 20))
Lemma 1. For a time domain De, the static causal factors can be obtained by solving the following objective:
where p( is presented a learnable categorical distribution. Together with the classification loss, the loss function for learning the evolving causal mechanisms is
max 4Ht
I(Y; tb (X1)),
s.t. 4et E arg max I ( (X:); (X.1)))),
of
mm
E Eyoavee Dogg(vel= dy znt 2xt,= z(]]
where maximizing the MI term can be achieved by mini- mizing the cross-entropy loss between the representation &c (X,,) and the target Y. Prop. 1 (ii) demonstrates that by using the static causal factors Z as the anchor, the dynamic causal representations can be learned through
(1I)
T
( (t
3.3.5. OVERALL OPTIMIZATION OBJECTIVE
max Φ dv
I dyy (X,); (X,); Y),, c
(8)
Let Levaive Lip Lmp, where Levalve represents the loss function for capturing the evolving
d patter. Let Lemual Lue + Laye, where Late - -i (t((t) and
where Φdy is comprised of the feature extraction component of qo, i.e. qott and neural-network-based masker may
Laye = - Σt= [ty(t), T (1) and Tay represent the term
6

---

Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
Table 1: Comparison of accuracy (%) between SYNC and other baseline methods. "Wst" and "Avg" denote worst-case and mean performance across all test domains for given dataset, respectively. The best and second best results are marked in bold and underline, respectively. "Overall" refers to the mean of the two metrics averaged across all datasets.
Circle
Sine
RMNIST
Portraits
Caltran
PowerSupply
ONP
Overall
Method
Wst
Avg
Wst
Avg
Wst
Avg
Wst
Avg
Wst
Avg
Wst
Avg
Wst
Avg
Wst
Avg
ERM
41.7
49.9
49.5
63.0
37.8
43.6
75.5
87.8
29.9
66.3
64.4
71.0
64.2
65.9
51.9
63.9
Mixup
41.7
48.4
49.3
62.9
38.3
44.9
75.5
87.8
52.3
66.0
64.3
70.8
64.1
66.0
55.1
63.8
MMD
41.7
50.7
47.6
55.8
39.0
44.8
74.0
87.3
29.4
57.1
64.8
70.9
47.1
51.3
49.1
59.7
MLDG
41.7
50.8
49.5
63.2
37.5
43.1
76.4
88.5
51.7
66.2
64.6
70.8
63.9
65.9
55.0
64.1
RSC
41.7
48.0
49.5
61.5
35.8
41.7
75.2
87.3
51.9
67.0
64.4
70.9
63.3
64.7
54.5
63.0
MTL
42.2
51.2
46,9
62.9
36,1
41.7
78.2
89.0
52.6
68.2
64.2
70.7
63.4
65.6
54.8
64.2
FISH
41.7
48.8
49.5
62.3
37.6
44.2
78.6
88.8
57.5
68.6
64.2
70.8
63.2
65.9
56.0
64.2
CORAL
41.7
53.9
46.3
51.6
38.5
44.5
74.6
87.4
50.9
65.7
64.6
71.0
63.8
65,8
54.3
62.8
AndMask
41.7
47.9
42,9
69.3
37.8
42.8
62.0
70.3
29.9
56.9
64.0
70.7
51.2
54.6
47.1
58.9
DIVA
56.7
67.9
38.6
52.9
36.9
42.7
76.2
88.2
53.8
69.2
63.9
70.7
64.5
66.0
55.8
65.4
IRM
41.7
51.3
49.5
63.2
34.1
39.0
74.2
85.4
43.1
60.6
64.1
70.8
61.9
64.5
52.7
62.1
IIB
42.0
53.9
47.6
61.3
38.5
45.0
78.1
89.7
54.8
69.3
64.5
70.8
63.2
66.3
55.2
62.1
IDAG
42.0
49.0
47.6
57.1
39.8
44.1
79.8
88.6
53.5
69.7
66.1
71.2
63.8
66.4
56.1
63.7
GI
42.0
54.4
49.8
65.2
39.2
44.6
77.8
88.1
53.1
70.7
65.1
71.0
63.5
65.7
55.8
65.7
LSSAE
42.0
73.8
49.0
71.4
40.3
46.4
77.7
89.1
54.3
70.3
65.4
71.1
64.5
66.0
56.2
69.7
DDA
42.0
51.2
43.0
66.6
38.0
45.1
76.0
87.9
31.0
66.1
64.4
70.9
63.5
65.3
51.2
64.7
DRAIN
45.0
50.7
43.0
71.3
37.2
43.8
77.7
89.4
55.7
69.0
64.9
71.0
59.8
61.1
54.8
65.2
MMD-LSAE
54.0
79.5
43.0
71.4
42.9
49.2
80.9
90.4
56.9
69.6
65.2
71.4
61.8
66.4
58.1
70.9
CTOT
54.0
75.2
43.2
67.2
31.7
44.8
79.2
86.4
48.0
66.9
63.6
71.1
61.1
65.6
54.4
68.2
SDE-EDG
45.0
81.5
42.3
72.2
39.7
52.6
78.6
89.6
55.1
71.3
64.1
70.8
63.1
65.6
55.4
71.9
SYNC (Ours)
67.0
84.7
60.0
76.1
45.8
50.8
81.0
90.8
58.8
72.2
66.9
71.7
64.6
65.6
63.4
73.1
at time as defined in Eq. (6) and Eq. (9), respectively. The objective function of SYNC is
4. Experiments
4.1. Experimental setup
LSYNC = Levotve + a1 CMI + Ceauall
(12)
Datasets. We evaluate SYNC on several commonly used benchmarks, including two synthetic datasets (Circle, Sine) and five real-world datasets (RMNIST, Portraits, Caltran, PowerSupply, ONP). Circle (Pesaranghader & Viktor, 2016) contains evolving 30 domains, where the instance are sam- pled from 30 2D Gaussian distributions. Sine (Pesarang- hader & Viktor, 2016) includes 24 evolving domains, which is achieved by extending and rearranging the original dataset. RMNIST (Ghifary et al., 2015) consists of MNIST digits with varying rotations. Following (Qin et al., 2022), we generate 19 evolving domains by sequentially applying ro- tations from 0 to 180 in 15" increments. Portraits (Yao et al., 2022a) is a real-world dataset of American high school senior photos spanning 108 years across 26 states. We split it into 34 evolving domains using fixed temporal intervals. surveillance dataset
where 01 and a2 are the trade-off hyperparameters.
3.4. Theoretical Analysis
In this section, we aim to provide theoretical insight on our proposed SYNC. More details on the theoretical proof and discussion can be found in Appendix B. First, we show that optimizing Levolve approximates the joint distribution p(Z1:T, )1.r) and captures the underlying evolving pattern. Theorem Assume that the underlying data generation process at each time step is characterized by SCM Mr- Then, by optimizing Levolve, the model can learn the data distribution p(21.T, V1:r) training domains.
Thm. demonstrates that minimizing Levolve can effectively capture the underlying evolving pattern in data distribution. Now, we provide Thm. 2 as follows to state that minimizing LSYNG leads to the optimal causal predictor at each time domain Dr, thereby addressing spurious correlations and improving model generalization.
Caltran (Hoffman et al., 2014) captured by fixed traffic cameras deployed at intersections and is split into 34 domains by time to predict the type of scene. PowerSupply (Dau et al., 2019) is created by an Ital- ian electricity company and contains 30 evolving domains. ONP (Fernandes et al., 2015) is collected from the Mashable website within two years and is divided into 24 domains according to month. All domains are split into source do- mains, intermediate domains and target domains according to the ratio of {1/2 1/6: 1/3). The intermediate domains
Let dbt (Xx), (X,) and Z denote the rep- resentations and drift factors the time domain Dr ob- tained by training the network through the optimization of EsYNC. Then the predictor constructed upon these compo- nents is the optimal causal predictor defined in Def. 2.
Theorem 2.
7

---

Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
30.0
90
independence
10.0
Independence
24.0
Accuracy(9)
90
7.5
Accura
85
5.0
12.0
50
MLDG
ERM
80
ERM
M0,3
DAG
2.5
6.0
52B EDG
KNDLSAA
75
S0e: +DG
VNG L54E
0.0
0.0
SYNC
SYNC
5
10
15 epochs
20
25
30
0
20
40
60 epochs
80
100
120
10
21
22
23
25 26 27 26 29 30 Domain Index
24
70 25 26
27 28 29 30 31 32 33 34 Domain index
(a) Circle
(b) RMNIST
(c) Circle
(d) Portraits
Figure 5: (a) and (b) show the independence degree of LSSAE and SYNC, respectively, during the training process on the Circle and RMNIST. (c) and (d) present the test accuracy trajectory of SYNC and various baselines on Circle and Portraits.
are utilized as validation set for model selection.
their generalization ability may deteriorate over longer tem- poral spans. In contrast, DG methods focus on learning domain-invariant representations, which can serve per- formance lower bound and offer more stable guarantee for generalization. These results demonstrate the impor- tance of leveraging stable representations to enhance model generalization in dynamic environments.
Baselines. We select three categories of related meth- ods for comparison: (i) Non-Causal DG: ERM (Vap- nik, 1999), Mixup (Zhang et al., 2018), MMD (Li et al., 2018b), MLDG (Li et al., 2018a), RSC (Huang et al., 2020), MTL. (Blanchard et al., 2021), FISH (Shi et al., 2022), CORAL (Sun & Saenko, 2016), AndMask (Paras- candolo et al., 2021), DIVA (Ilse et al., 2020). (ii) Causal DG: IRM (Arjovsky et al., 2019), IIB (Li et al., 2022), iDAG (Huang et al., 2023). (iii) EDG: GI (Nasery al., 2021), LSSAE (Qin et al., 2022), DDA (Zeng et al., 2023b),
SYNC consistently outperforms other baselines over all benchmarks, achieving an accuracy of 63.4% in the worst- case scenario and 73.1% in terms of average performance. Specifically, compared to existing causal DG methods, SYNC achieves significantly better results on both metrics, with improvements of at least 7.6% in worst-case perfor- mance and 7.7% in average performance. This highlights the importance of incorporating dynamic causal information and modeling causal mechanism shifts. Moreover, when compared with EDG methods, SYNC outperforms the state- of-the-art approach by 5.3% and 1.2% in the two metrics, respectively. These results demonstrates that, by learning the causal model, SYNC achieves superior generalization and captures more robust evolving patterns. Lastly, it can be observed that VAB-based methods, such as LSSAF,, achieve better performance, suggesting that generative approaches are effective in capturing more robust evolving patterns. Building on this, SYNC further learns both static and dy- namic causal representations, which enhances model gener- alization and advances the modeling of evolving patterns.
DRAIN (Bai et al., 2023), CTOT (Jin et al., 2024). SDE- EDG (Zeng et al., 2023a), MMD-LSAE (Qin et al., 2023).
We keep the neural network architecture of encoding and classification part constant across all baselines used in dif- ferent benchmarks for fair comparison.
Implementation. All experiments in this work are per- formed on single NVIDIA GeForce RTX 4090 GPU with 24GB memory, using the Py PyTorch packages, and are based on DomainBed (Gulrajani & Lopez-Paz, 2021). Please refer Appendix E.2 for additional training details and Ap- pendix C.3 for network architecture details.
Evaluation Metries. In this work, we report the gen- eralization performance on K target domains in the fu- ture, including the worst accuracy performance "Wst" (minke _ Accuracy(DD +k)) and the average accu- racy performance "Avg" (+ Σk Accuracy(DT+.))
4.3. Analytical Experiments
4.2. Quantitative results
Ablation study We conduct an ablation study on RM- NIST to evaluate the effectiveness of various components in our method, and results are presented in Table 2. Vari- ant A serves as the base method trained solely with evolv- ing pattern loss Levalve Variant B builds upon the base method by additionally training with MI loss LMI. can be observed that Variant B achieves better performance than Variant A, suggesting that better separation enables more effective utilization of features from each part. Variants C and D build upon Variant B by incorporating additional training with static causal loss Lae and dynamic causal
The results of our proposed SYNC, along with various base- line methods, are presented Table 1. It can be observed that the average performance of EDG methods generally exceeds that of traditional DG methods, underscoring the importance of modeling and leveraging evolving patterns in dynamic environments. In terms of worst-case performance, EDG methods do not exhibit a significant improvement over DG methods, in contrast to their advantage in aver- age performance. This may be attributed to the fact that, if EDG methods fail capture robust evolutionary patterns,
8

---

Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
5
4
3
2
2
0
0
0
-1
-1
-1
2
5
2
(a) ERM
(b) MMD-LSAE
(c) SYNC
(d) Ground truth
Figure 6: Visualization of decision boundary of the Circle dataset, where the positive and negative classes are colored in red and blue. (a)-(c) show the decision boundaries learned by ERM, MMD-LSAE and SYNC. (d) presents the ground truth.
Table 2: Ablation study of SYNC on RMNIST dataset.
methods on Circle and Portraits. SYNC demonstrates more stable predictive performance while maintaining high accu- racy. Notably, compared to SDE-EDG on the Circle dataset, SYNC continues to perform well in the later domains, high- lighting the effectiveness of time-aware causal representa- tion learning in enhancing long-tenn nerg-term&n generalization.
Ablation
Evo.
Disent.
z
zdy
Wst
Avg
Overall
Variant
40.5
44.1
42.3
Variant
/
41.9
45.7
43.8
Variant C
v
v
44.1
48.7
46.4
Variant
42.9
49.2
46.1
SYNC
/ v
45.8
50.8
48.3
Evaluation on evolving pattern captured In order to verify whether the evolving patterns learned through causal representations are more accurate, we visualize the decision boundaries of our proposed approach along with ERM and MMD-LSAE on the Circle dataset. As shown in Fig. target domains due
loss Laye, respectively. Variant C and D achieve higher curacy than Variant B, demonstrating that learning causal representations enhances model generalization. Notably, it is clear that Variant C achieves greater improvement in worst-case performance compared to Variant D, indicating that static causal factors can ensure stable generalization under continuous distribution shifts. However, focusing solely on them ignores evolving pattern, limiting further generalization gains. Learning dynamic causal factors cap- tures features related to task evolving over time, enabling to generalize better to the current distribution. Variant D outperforms Variant in average performance and provides evidence for this claim. SYNC jointly learns static and dynamic causal representations and achieves the best perfor- mance, highlighting their significant contributions overall effectiveness. This demonstrates that static and dynamic causal factors are complementary and play crucial role in enhancing model generalization.
ERM struggles generalize to to its inability model latent evolving patterns. In contrast, MMD-L.SF demonstrates improved generalization to fu- ture domains. SYNC goes one step further and achieves decision boundaries that most closely align with the ground truth, highlighting effectiveness capturing evol ving patterns and enhancing generalization unseen target data.
5. Conclusion
In this work, we investigate the issue that existing EDG methods may experience poor generalization performance due to the presence of spurious correlations. To address this problem, we propose time-aware SCM by considering both dynamic causal factors and causal mechanism drifts. We propose Static-DYName Causal Representation Learning (SYNC), approach that effectively learn time-aware causal representations and reconstruct causal mechanisms, thereby obtaining the causal model. Extensive experiments demonstrate the effectiveness and superiority of our method in improving the temporal generalization performance. hope this work can offer valuable insights into improving model generalization in dynamic environments.
To illustrate the extent of disentangle- ment between static and dynamic factors in our approach, we use mutual information as the measure independence and estimate using mini-batch weighted sampling between the static and dynamic representations produced by LSSAE and SYNC on the Circle and RMNIST datasets. As shown in Fig. 5a and Fig. 5b, compared to LSSAF, SYNC exhibits faster and more stable decline in the independence indi- cator, indicating that our method achieves a more effective disentanglement of static and dynamic factors.
Disentanglement
A Acknowledgements
This paper was supported by the National Natural Science Foundation of China (No. 62376026), Beijing Nova Pro- gram (No. 20230484296) and KuaiShou.
Temporal Robustness Fig. 5c and Fig. 5d show the ac- curacy trajectories of our method and multiple baseline

---

Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
Impact Statement
Chen, J., Song, L., Wainwright, M., and Jordan, M. Learn- ing explain: An information-theoreti perspective model interpretation. In ICML, pp. 883-892, 2018a.
In this work, we highlight the potential negative impact of spurious correlations on the model generalization in the evolving domain generalization problem and provide in- sights into how to build and learn causal models in non- stationary environments. By learning time-aware causal representations, the model can effectively mitigate the im- pact of task-irrelevant factors in dynamic tasks, thereby enabling reliable predictions and secision-making in com- plex, changing environments. This allows the model to cope with challenges in non-stationary dynamic environments, offering potential benefits for real-world applications such autonomous driving and advertisement recommendation.
Chen, Gao. Z., Wu, X., and Luo, Meta-causal learning for single domain generalization. In CVPR, pp. 7683- 7692, 2023.
Chen, R. T., Li, X., Grosse, R. B., and Duvenaud, D. K. Isolating sources of disentanglement in variational au- toencoders. NeurlPS, 2018b.
Chen, Y., Zhang, Y., Bian, Y., Yang, H., Kaili, M., Xie, B., Liu, T., Han, B., and Cheng, Learning causally invariant representations for out-of-distribution generalization graphs. NeurlPS, pp. 22131-22144 2218,, 2022.
References
Dau, A., Bagnall, A., Kamgar, K., Yeh, C.-C. M., Zhu, Y., Gharghabi, S., Ratanamahatana, C. A., and Keogh, The ucr time series archive. JEEE/CAA JAS, pp. 1293- 1305, 2019.
Ahuja, K., Shammugam, K., Varshney, K., and Dhurandhar, A. Invariant risk minimization games. In JCML, pp. 145-155, 2020.
Ahuja, K., Caballero, E., Zhang, D., Gagnon-Audet, J., Bengio, Y., Mitliagkas, I., and Rish, I. Invariance princi- ple meets information bottleneck for out-of-distribution generalization. In NeurlPS, pp. 3438-3450, 2021.
Entner, and Hoyer, P. O. On causal discovery from time series data using fci. Probabilistic graphical models, 2010.
Fan, S., Pagliardini, M., and Jaggi, M. DOGE: domain reweighting with generalization estimation. In ICML, 2024.
Arjovsky, M., Bottou, L., Guirajani, I. and Lopez- Paz, D. Invariant risk minimization. arXiv preprint arXiv: 997.02893, 2019.
Fernandes, K., Vinagre, P., and Cortez, P. A proactive intelli- gent decision support system for predicting the popularity of online news. In Prog. Artif. Intell., pp. 535-546, 2015.
Arpit, D., Wang, H., Zhou, Y., and Xiong, C. Ensemble of averages: Improving model selection and boosting performance in domain generalization. In NeurlPS, 2022.
Gama, J., Žliobaité, I., Bifet, A., Pechenizkiy, M., and Bouchachia, A. A survey on concept drift adaptation. ACM computing surveys (CSUR), (4):1-37, 2014.
Bai, G., Ling, C., and Zhao, L. Temporal domain gener- alization with drift-aware dynamic neural networks. In ICLR, 2023.
Ghifary, M., Kleijn, W. B., Zhang, M., and Balduzzi, D. Domain generalization for object recognition with multi- task autoencoders. In ICCV, pp.
Balaji, Y., Samkaranarayanan, S., and Chellappa, R. Metareg: Towards domain generalization using reta-regullriization. In NewrIPS, pp. 1006-1016, 2018
2015.
Ghifary, M., Balduzzi, D., Kleijn, W. B., and Zhang, M. Scatter component analysis: unified framework for domain adaptation and domain generalization. TPAMI, pp. 1414 1430, 2017.
Belghazi, M. I., Baratin, A., Rajeshwar, S., Ozair, S., Ben- gio, Y., Courville, A., and Hjelm, D. Mutual information neural estimation. In ICML, pp. 531-540, 2018.
Grossniklaus, E., Nickerson, J. M., Edelhauser, H. F., Bergman, L. A., and Berglin, L. Anatomic alterations in aging and age-related diseases of the eye. IOVS, 54(14): ORSF23-ORSF27, 2013.
Blanchard, G., Deshmukh, A. A., Dogan, U., Lee, G., and Scott, C. Domain generalization by marginal transfer learning. JMLR, pp. 1-55, 2021.
Carlucci, M., D' I'Innocente, A., Bucci, S., Caputo, B., and Tommasi, T. Domain generalization by solving jigsaw puzzles. In CVPR, pp. 2229-2238, 2019.
Gulrajani, I. and Lopez-Paz, D. search of lost domain generalization. ICLR, 2021.
Hendrycks, D.,, Mu, N., Cubuk, E. D., Zoph, B., Gilmer, J., and Lakshminarayanan, B. Augmix: A simple data processing method to improve robustness and uncertainty. ICLR, 2020.
Chattopadhyay, P., Balaji, Y., and Hoffman, J. Learning to balance specificity and invariance for in and out of domain generalization. In ECCV, pp. 301-318, 2020.
10

---

Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
Hochreiter, S. and Schmidhuber, Long short-term memory. Neural Comput., 9(8):1735-1785 1997.
Liu, J., Shen, Z., He, Y., Zhang. X., Xu, R., Yu, H., and Cui, P. Towards out-if-dist generalization: A survey. arXiv preprint arXiv:2108. 13324 2021b.
Hoffman, J., Darrell, T., and Saenko, K. Continuous man- ifold based adaptation for evolving visual domains. In CVPR, pp. 867-874, 2014.
Lu, C., Wu, Y., Hernández-Lobat, J. M., and Schölkopf, В. Invariant causal representation learning for out-of- distribution generalization. In ICLR, 2021.
Hu, Y., Jia, X., Tomizuka, M., and Zhan, W. Causal-based time series domain generalization for vehicle intention prediction. In ICRA, pp. 0006- 2022.
Lu, J., Liu, A., Dong, F., Gu, F., Gama, J., and Zhang,
G. Learning under concept drift: A review. TKDE, pp. 2018.
Huang, Z., Wang, H., Xing, E. P., and Huang, D. Self- challenging improves cross-domain generalization. In ECCV, pp. 124-140, 2020.
Lv, F., Liang, J., Li, S., Zang. B., Liu, C. H., Wang, Z. and Liu, D. Causality inspired representation learning for domain generalization. In CVPR, pp. 8046-8056, 2022.
Huang, Z., Wang, H., Zhao, J., and Zheng, N. idag: Invariant dag searching for domain generalization. In ICCV, pp. 19169-19179, 2023.
Mahajan, D., Tople, S., and Sharma, A. Domain general- ization using causal matching. In ICML, pp. 7313-7324, 2021.
llse, M., Tomezak, J. M.,, Louizos, C., and Welling, M. Diva: Domain invariant variational autoencoders. In MIDI. pp. 322-348, 2020.
Malinsky, D. and Spirtes, P. Causal structure learning from multivariate time series in settings with unmeasured con- founding. In Proceedings of 2018 ACM SIGKDD work- shop causal discovery, pp. 23 47, 2018.
Jang, E., Gu, S., and Poole, B. Categorical reparameteriza- tion with gumbel-softmax. ICLR, 2017.
Jin, Y., Yang, Z., Chu, X., and Ma, L. Temporal domain gen- cralization via learning instance-level evolving patterns. In IJCAI, pp. 4255-4263, 2024.
Mao, C., Xia, K., Wang, J., Wang. H., Yang. J., Bareinboim, E., and Vondrick, C. Causal transportability for visual recognition. In CVPR, pp. 7521-7531, 2022.
Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., and Krishnan, D. Supervised contrastive learning. NeurlPS, pp. 18661-18673, 2020.
Motiian, S., Piccirilli, M., Adjeroh, D. A., and Doretto, G. Unified deep supervised domain adaptation and general- ization. In ICCV, pp. 7776, 2017.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In ICLR, 2014.
Muandet, K., Balduzzi, D., and Schölkopf, B. Domain generalization via invariant feature representation. In ICMI,, pp. 10-18, 2013.
Li, B., Shen, Y., Wang. Y., Zhu, W. Li. D., Keutzer, K., and Zhao, H. Invariant information bottleneck for domain generalization. In AAAI, pp. 7399-7407, 2022.
Nasery, A., Thakur, S., Piratla, V., De, A., and Sarawagi, S. Training for the future: A simple gradient interpolation loss to generalize along time. In NeurlPS, pp. 19198 19209, 2021.
Li, D., Yang. Y., Song. Y.-Z., and Hospedales, T. Learning to generalize: Meta-learning for domain generalization. In AAAI, pp. 3490-3497, 2018a.
Oord, A. v. d., Li, Y., and Vinyals, O. Representation learn- ing with contrastive predictive coding. arXiv preprint 03748, 2018.
Li, H., Pan, S. J., Wang, S., and Kot, A. C. Domain gener- alization with adversarial feature learning. ECCV, pp. 5400-5409, 2018b.
Li, H., Pan, J., Wang. S., and Kot, A. C. Domain gener- alization with adversarial feature learning. In CVPR, pp- 5400-5409, 2018c
Parascandolo, G., Neitz, A., Orvieto, A., Gresele, L., and Schölkopf, B. Learning explanations that are hard to vary. In ICLR, 2021.
Li, J., Wu, B., Sun, X., and Wang, Y. Causal hidden markov model for time series disease forecasting. In CVPR, pp. 12105-12114, 2021.
Pearl, J. Causality. Cambridge university press, 2009.
Pearl, J., Glymour, M., and Jewell, N. P. Causal inference in statistics: A primer. 2016.
Liu, C., Sun, X., Wang, J., Tang, H., Li, T., Qin, T., Chen, W., and Liu, T. Learning causal semantic representation for diitr---iitiitttinn prediction. In NeurIPS, pp. 6155- 6170, 2021a.
Peng, X. and Saenko, K. Synthetic to real adaptation with generative correlation alignment networks. In WACV, pp- 1982-1991, 2018

---

Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
Pesaranghader, A. and Viktor, H. L. Fast hoeffding drift detection method for evolving data streams, In ECML/PKDD, pp. 96-111, 2016.
Xie, B., Chen, Y., Wang. J., Zhou, K., Han, B., Meng, W., and Cheng. J. Enhancing evolving domain generalization through dynamic latent representations. In AAAI, pp. 16040-16048, 2024.
Piratla, V., Netrapalli, P., and Sarawagi, S. Efficient domain generalization via common-specific low-rank decomposi- tion. In ICML, pp. 7728-7738, 2020.
Xu, Q.. Zhang, R., Zhang, Y., Wang, Y., and Tian, Q. A fourier-based framework for domain generalization. In CVPR, pp. 14383-14392, 2021.
Qin, T., Wang, S., and Li, H. Generalizing to evolving do- mains with latent structure-aware sequential autoencoder. In JCML. pp. 18062-18082. 2022.
Yao, H., Choi, C., Cao, B., Lee, Y., Koh, W., and Finn, Wild-time: A benchmark of in-the-wild distribution shift over time. In NeurIPS, 2022a.
Qin, T., Wang. S., and Li, H. Evolving domain general- ization via latent structure-aware sequential autoencoder. TPAMI, 2023.
Yao, W., Chen, G., and Zhang, K. Temporally disentangled representation learning. In NeurIPS, pp. 26492-26503, 2022b.
Recht, B., Roelofs, R., Schmidt, L.,, and Shankar, V. Do imagenet classifiers generalize to imagenet? In ICML, pp. 5389-5400, 2019.
Yong. L., Zhou, F., Tan, L., Ma, L., Liu, J., He, Y., Yuan, Y., Liu, Y., Zhang, J. Y., Yang, Y., and Wang, H. Continuous invariance learning. In ICLR, 2024.
Sheth, P., Moraffah, R., Candan, K. S., Raglin, A., and Liu, H. Domain generalization-a causal perspective. arXiv preprint wwXv:2209.11177. 2022.
Yue, Z., Sun, Q., Hua, X.-S., and Zhang, H. Transporting causal mechanisms for unsupervised domain adaptation. In ICCV, pp. 8599-8608, 2021.
Shi, Y., Seely, J., Torr, P. H. S., Narayanaswamy, S., Hannun, A. Y., Usumier, N., and Synnaeve, G. Gradient matching for domain generalization. In ICLR, 2022.
Zeng, Q., Shui, C., Huang. L.-K., Liu, P., Chen, X., Ling, C., and Wang, B. Latent trajectory learning for limited timestamps under distribution shift over time. In ICLR, 2023a.
Sun, B. and Saenko, K. Deep coral: Correlation alignment for deep domain adaptation. In ECCV, pp. 443-450, 2016.
Zeng, Q., Wang, W., Zhou, F., Ling, C., and Wang, B. Fore- see what you will learn: data augmentation for domain generalization in non-stationary environment. In AAAI, pp. 11147-11155. 2023b.
Sun, X., Wu, B., Zheng, X., Liu, C., Chen, W., Qin, T., and Liu, T. Recovering latent causal factor for generalization to distributional shifts. In NeurlPS, pp. 16846-16859, 2021.
Zeng, Q., Wang. W., Zhou, F., Xu, G., Pu, R., Shui, C., Gagné, C., Yang. S., Ling, C. X., and Wang. B. General- izing across temporal domains with koopman operators.
Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., and Schmidt, L. Measuring robustness to natural distribution shifts in image classification. In NeurIPS, pp. 18583- 18599, 2020.
In AAAI, pp. 16651-16659, 2024.
Zhang, H., Cissé, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. In ICLR, 2018.
Vapnik, V. N. An overview of statistical learning theory. IEEE Trans. Neural Netw., pp. 988-999, 1999.
Zhou, K., Yang. Y., Hospedales, T. M., and Xiang. T. Learn- ing generate novel domains for domain generalization. In ECCV, pp. 561-578. 2020.
Wang, J., Lan, C., Liu, C., Ouyang, Y., Qin, T., Lu, W., Chen, Y., Zeng, W., and Yu, P. S. Generalizing to unseen domains: survey domain generalization. TKDE, pp- 8052-8072. 8072, 2023.
Zhou, K., Yang, Y., Qiao, Y., and Xiang, T. Domain gener- alization with mixstyle. In ICLR, 2021a.
Zhou, K., Yang, Y., Qiao, Y., and Xiang, T. Domain adaptive ensemble learning. TIP, 30:8008888-818 8018, 2021b.
Wang, R., Yi, M., Chen, Z., and Zhu, S. Out-of-distribution generalization with causal invariant transformations. In CVPR, pp. 375-385, 2022.
Wang, S., Zhu, J., Wang, Y., Ma, C., Zhao, X., Zhang, Y., Yuan, Z., and Ruan. S. Hierarchical gating network for cross-domain sequential recommendation. TOIS, 2025.
12

---

Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
A. Related Work
Domain Generalization (DG) aims to generalize knowledge extracted from multiple source domains to unseen target domain. recent years, extensive research has been conducted to address this issue in the literature (Wang al., 2023; Liu et al., 2021b), which can be roughly categorized into three groups: (i) Invariant representation learning: This category of methods focus learning invariant representations across distributions to generalize well to unseen target domains, which is achieved through domain alignment (Muandet et al., 2013; Ghifary et al., 2017: Li et al., 201 2018c) and feature disentangement (Piratla et al., 2020; Chattopadhyay et al., 2020). Domain alignment methods learn invariant representation via kernel-based optimization (Muandet et al., 2013; Ghifary et al., 2017), adversal training (Li et al., 10; Motiaan et al., 2017) and using additional normalization (Peng & Saenko, 2018). Feature disentaglement methods (Piratla et al., 2020; Chattopadhyay al., 2020) aim disentagle the features into domain-specific and domain-invariant features, and the latter are utilized for better generalization. (ii) Domain augmentation: These methods enrich the diversity of the training domain by manipulating domain data, thereby enhancing the model's generalization. They encompass operations at both the image levels (Zhang al., 2018; Hendrycks et al., 2020) and feature levels (Zhou et al., 202 2020; Xu et al., 2021). (iii) Learning strategies: This category of methods utilize effective learning strategies to improve model generalization performance, such as meta-learning (Balaji et al., 2018; Li et al., 2018a), ensemble learning (Zhou et al., 2021b; Arpit et al., 2022) and seff-vuper learning (Carlucci et al., 2019; Huang et al., 2020). Although significant progress has been made in DG, when faced with non-staiie -tasstattonay environments that are more common in the real world (Grossniklaus al., 2013; Wang et al., 2025), still faces major challenges (Yao et al., 2022a; Qin et al., 2022).
B. Theoretical Details
B.1. Proof of Proposition 1
Proposition 1. Let De be a time domain, for given class Y, it can be conclude that:
(i) If there is HZ Zst 2,, ( " then 1 1(Zg) Zst 2,,--1 > 1 8,85 2zt 2
(i) If there is H HZat zdy, Y) < ( 1 Y), then T( 2
T(Zdy: 2.2: Zet V)
Proof. For [(; YY) according to the properties of conditional MI, it can be derived that
[[:
(13)
I(: (() = -
thus we have
H 1Z Y) H(Z [,,
( = I(Zef; - |Y) > I(Z7,, - l8),
(14)
H (2 c,1 dy Y) < H1 (2225 /dy Y) + 1(2dy, 2st Y > 1 (2224 (.)
These two propositions regarding the properties of causal factors demonstrate that, as long as the causal factors are sufficiently similar, the conditional MI can be maximized to distinguish the causal factors from the spurious factors.
B.2. Proof of Theorem
Theorem 1. Assume that the underlying data generation process at each time step is characterized by SCM My. Then, by optimizing Levolee the model can learn the data distribution p(æ1T, DrrT) on training domains.
13

---

Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
Proof. According Eq. (1) and Eq. (11), minimizing Levolve is equivalent to maximizing
T
T
T
- Levolve
E qu [logp(ze(<" 2
Σ
DKL xt), - Σ
DxL a.), 12
==1
Boog logp(v/>=
dy Raz 24)]
Σ
(15)
The above equation can be further simplified to
Levolve
E Roo [log p20 , 2t 20
Σ E
Boo [logp(ueee 12 =dy 24)
Ruga log (2 (22)
T
dy Egg log go(2)
T
Ic (<(= 2= Vt)
Σ
Σ
Egg log p(=
(16)
p p(z7t)
p(zz 22 ==(()
T
pl 12
22 du )p((t Zo.t Zo,t:
dy 24 )p)pp( (p (11 )'pp(((
dy du
(2))p(z 1=(1)
Begg log
qv (x )o (z: dy >>> az ) =
Based on the twme-aware SCM MT and the temporally evolving SCM Menan and let q(z)" =z 2, Ut) qv (2 (2x))0 dy x (2))((21)2 y1), then it can be obtained that
st z az 2 2t dy )p(y
pp( 1 (<)))((()) [
Levolve
Σ
go log
(x xx)) ge dy dy
)cc
3/.)
p( st
du plUi Rel
dy
dy dy
S Equgogs
log
q (z x g0
(17)
1= p( 11 2 (V c. zc, 21 )p( 2 du 2 )p) (z(
=Egugog log
I gv (2 (2 x 40 2, 2 a: )g (2 2
dy
p( 2i. = 1.7 zir)
= Byg log
q (z
2..
=
1 1.T W1.T)
Since the distribution of observational data p(æ1:1, Mirr) is generated by the latent factors Ze Zdy and Zd, aim to model this distribution by approximating the posterior pl (( 2 =f. laner Vhr) with a variational network q:
Dxr(g.p) = =E, log q 2 1.T
dy dd 11.T, (I.T)
p(z - 2 (.:r)
(18)
p 2 1: 2 1.T #1T. + log
dy
ER, log
g( 1:T, i 1:T (1IT,
dy d
Combing with Eq. (17), it is clear that optimizing Levolve is equivalent to minimizing T p), which facilitates learning the data distribution p( Bur)).
B.3. Proof of Theorem 2
In this section, we elaborate how SYNC derives the optimal causal predictor for each time domain. We first briefly introduce the relevant theories of causal graphical models and causal assumptions in Sec. B.3.1. Then, we provide the proofs of Lem. 1 and Lem. 2 in Sec. B.3.2 and Sec. B.3.3, respectively. Finally, the proof of Thm. 2 is presented in Sec. B.3.4.
B. PRELIMINARIES
Causal Graphical Models. In causal research (Pearl, 2009; Pearl et al., 2016), a causal graphical model & is a directed acyclic graph (DAG) that can be derived from SCM M, characterizing the joint probability distribution P over n random
14

---

Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
variables X1, X2.... x.- The causal DAG g is comprised of the nodes V and the directed edges E, where the starting point and end point of each directed edge (-+ represent cause and effect, respectively. The directed edges encode the causal relationships between the variables. The joint distribution P has causal factorization as follows:
a P(X1st)= = II P(X.)PA(X)),
(19)
where (X;; denotes the parents of variable X; in DAG G. Causal graphical models typically consist of three fundamental structures: chain, fork, and collider. The three basic causal structures and their metaphorical dependencies are as follows:
Chain: X Y - Z. In chain structure, X is cause that influences Y, and Y is a cause of Z. This structure implies dependency: X IL Z, while X / Y.
Fork: X <- Y + Z. In this structure, Y is the common parent of both X and Z, and is therefore referred to confounder. A fork structure shows that X A Z, while X 1L Z|Y.
Collider: X - < Z. Y is the common child of the two variables, X and Z, and is referred to as a collider. The dependency implied by collider structure is X IL Z, while X y Z
Causal Assumptions. Due to the available data offer only partial insights into the underlying causal mechanisms. Therefore, it is crucial to make certain priors assumptions about the structure of the causal relationships enable causality learning. The following assumptions (Pearl, 2009; Pearl et al., 2016) are commonly used:
Causal Markovian Condition. For each variable in causal model, if given its parents, then it is conditionally independent of all its non-descendants.
Causal Sufficient. The causal sufficiency assumption states that there are no unmeasured common causes of any pair of variables in causal model. This assumption is crucial in wide range of literature.
Global Markov Condition. Given a DAG G and the joint distribution P of all nodes, for any two non-adjacent nodes X, and X, in G. and given a set of nodes Z, if X, and X, are d-separated by Z. then X, IL X, Z. We say that the distribution P satisfies the global Markov property with respect to the DAG G.
Causal Faithfulness. Given a DAG G and the joint probability distribution P of all nodes, for any two non-adjacent nodes X, and X, in G, given set of nodes Z, if X, 1L X, Z, then the nodes X, and X, are d-separated by the node set Z. We say that the distribution P satisfies the faithfulness with respect G. This assumption represents exactly the distributional independence relations implied by d-separation.
With these theories, combine the time-aware SCM Mr give the following proposition, which leads the definition of the optimal causal predictor.
Proposition 2 (Conditional Independence). It is assumed that there unmeasured common causes of any pair of variables in Mr, i.e., MM is causal sufficient. With the global Markov condition and the causal faithfulness, the conditional independence of latent factors can be derived from d-separation: (i) Y A [Z Zdy] [z Zdy]. (ii)
Y 1L 1Z 7dy [Z 7dy Zz)
Proof. The DAG corresponding to the time-aware SCM My is shown in Fig. 2b. It can be observed that there exists three paths from (Z Zd to Y: 2 G + Z + Y, Zgu + L + z + Y, and Zdyy + - 2d + Y. Note that the causal sufficiency of Mr guarantees that there are no other unobserved variables, thus all the paths between (Z.r, Z and are blocked by z Zdy and 2dd Therefore, it follows that Z Zdd and 2d block all the paths from T to Y. whereas Zer and 7 do not. According to d-separation criterion, it is obtained that Y A MM [Z Z [Z Zdd Y IL Mr [2: Zdd] 12 2uy 2d). Since the global Markov condition and the causal faithfulness guarantee the equivalence between d- separation and conditional independence, therefore we have:
Y A |Z ~.4 1Z 2 Y II [2 7.4y
1 7ul 774).
(20)
15

---

Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
From Prop. 2, it follows that for each time domain Dr, if we can obtain the static causal factors Z dynamicc causal factors 7gy, and drift factors Zd that indicate the state of the causal mechanism in the current time domain, then can use them to construct stable causal relationship between features and labels. The predictor built in this way will not be influenced by spurious factors. Additionally, the ideal causal features should capture as much information as possible about the target. Based on the above analysis, we now define the optimal causal predictor.
Definition 2 (Optimal Causal Predictor). For a time domain Dt, the corresponding drift factors 2 are given. Let Ze. (Z Zelt) be the causal factors in domain Dj that satisfy Ze E arg maXz.. I(Y; Ze,t Zτ¹). and Y IL [Z Z [Zo,t, ] Then the predictor based on factors Zo and 2 is the optimal causal predictor.
The following two lemmas establish a theoretical foundation for learning the causal factors Ze,
B.3.2. PROOF OF LEMMA 1
Lemma 1. For a time domain Dr. the static causal factors can be obtained by solving the the following objective:
max 4-
IY; 22 (X)). s.t. t arg max (Xt): 2 (Xi-)))))
(21)
Proof. For simplicity, use T.t represent T (X,). For pair of domains (.) sampled from training domains {DD))=1: maximizing conditional MI ( T -V) is essentially maximizing T( L; - L - I (Chen et al., 2022), where and 1 represent all the information within the corresponding time domain (including both global and local information) and serve as proxy variables for time. Thus, we have
c.t:
max ((0 ". (x(()))
max I( o,t T 1c--1,-
(22)
max 45
H( 2,t (V) H( at 21-11 -
Let Zedt and Z denote the ground truth static causal factors and static spurious factors, respectively, while Z C Z and Zis C Zed represent the static causal factors and static spurious factors learned by the model. After that, we define 74t. Zet Z and 7 7 Z the remaining static causal factors and static spurious factors. For term
H ( c, t|),, according to chain rule of entropy:
H c,, (V) = H(MY) + H( 14, Y) H(
(23)
where the second equality is due to f is determined so that (tlr) = 0. Suppose that the representations t learned by the model contain both causal information and spurious information, i.e., Th (Zie, Z(r,),, then Eq. (23) can be rewritten as
H YY = HZZ,, (t, H (Zi,, Y) + H (Z t,
(24)
Analogously, replacing be with the ground truth Z (Z (Z ) yields
H Y) = H(Zie,, 14, Y) = H H (Z 11, Y) + H (2 12 L Y).
(25)
Hence, it can be konwn that
AH (., = H( |t, Y) H( ()
= (H(Zio, |t, Y) + H (Z 1, t, x)) - (H Y)+ H 12
12 Y))
(26)
HZ 2ee, t, Y) (2 12, t,,
For the second term H ($at // a/ c,1-1, 1) in Eq. (22), suppose that Tel, (Zit,t Zgh,), Tst (Zi,,t11> 1,, then it can be derived that
H($ at Dat
ot 1 1, = (Zi', 2i,, 1/ 7 Zi, 1,
= -( 1/ Z , zet - - 1,Y) + H H( Z 1 2 7st I1,,t-1, 1,Y)
(27)
16

---

Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
Similarly, let c,t-1 (e, 2re, Zpc,11)) can get:
H(Z,, t 2 11-11 t- - 1, 1, = H (Zi,,, t| Z Zie,t- 1, +
H( -1.,,t--1,)) 1,
Zto, t, Zie, Zic,t-1. Z re, 1,, -,
(28)
Combing with Eq. (27) and Eq. (28), we have:
AH( c. Tet c1-1, 1- 1, H( 1 1, H( c,t: (/0
= H(iie, 11 7 Z 11, T / 1, - H(( 11 1/755, 7 7st
lc,t-11 12,--1: 1
1,
+ H 7 t1 L, Zie, -e,t-1 7 fr,tt-1, 1, - H /1o, Zie, -e 7
11,,1-11 1 1, 1, (29)
Now we can derive the expression for the change in the conditional MI [( c,fi 1,,-1|1):
I [Z - - I(
c,6i Dor
AI c,15 Det 2.1-1
AH Y) AH ( AH(HH, 0111::-1,,)
= (H( Zic,0 (,
H(Z /Te,
17,,, 1 7 1 7 1-, 1,
(30)
+ (H 1.:e:4424 17,6,1 2
1e 11, 1, 1, H(Zie, 11 1l7,e,1 Zie.- Ze, 1,
Zet
term 2
(H Lintt L, 2le
11,, 1,
H ( 2i
27c (Y
term 3
For term in Eq. (30), since conditioning reduces the entropy for both discrete and continuous variables, we can conclude that the value of term is greater that For term 2, since the similarity between causal factors is generally higher than the similarity between causal factors and spurious factors, the entropy of the second half is smaller than that of the first half, Therefore, the value of term 2 is greater than O. For term 3, it be rewritten -[(Zig,e Zis,--:: Zic,t-1 L 1 1|Zić, I, Y). When t is used condition, it is equivalent to giving all the information of D,. At this time, the spurious factors at time t and the factors at time f 1 are independent of each other, so the value of term 3 is O. Based on the above analysis, we have
c 0. T( 2=t c, (Y) T(
c,t Tht T
(31)
The above inequality indicates that maximizing the conditional MI 1( 3 t - causal factors Z Hence, Lem. is proved.
will allow & (.) to capture static
We have known that (-) can extract dynamic category-related information by optimizing the cross-entropy loss between Φdy (X) and target Y. The following Lem. 2 shows that dynamic causal factors Z can be learned by introducing the intra-domain conditional MI constraints.
B.3.3. PROOF OF LEMMA 2
Lemma 2. For time domain Dr, let (Xi) be the static causal representations learned by the model, then the dynamic causal factors of X₁ can be obtained through the following objective:
max I(Y; dud (x)(())) s.t. ddy E aarg max (X)) 4/6
(32)
Proof. For simplicity, we use qst, and dyy to represent T5d (X,) and (X,), respectively. Let Zuly and Z be the oft ground truth dynamic causal factors and dynamic spurious factors, respectively, while zdy c Zdy and z Z represent the dynamic causal factors and dynamic spurious factors learned by the model. After that, we define Zdy = Z zdy and zuy Zdu Zide the remaining dynamic causal factors and dynamic spurious factors. For I(Tay. Vel. (Y), suppose that the representations dd learned by the model contain both causal information and spurious information, i.e. dy ctt = (Z/g 1, then we rewrite as:
I(Tdy, st (y) T(ZLe / 7 VotlV)= H(Z Zuu + H(Zat
H(Z Zdy 7 Vo 1o,t
(33)
ot
17

---

Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
Analogously, 1( & eet (Y) (Y)
is expanded as:
I( I( (x)
/dy 2r,,11) H(Z H H( Z (erl H H(
(34)
Hence, we have:
A o1 (rr1"r)))
1 = I(Z (2 - I( c.t. (x)
[1(Z)e ( - H (Z ((od)r)
(H(Zee Lee, Zailr) ) - H H
(Z s0,e3
1o, Ze,1
AH,
AH2
(35)
For AH1, we have:
AH = H(Z Z - H( (Z Zie,c (
(H(Z (Y) + H Zi Z 1c,t Y) - (( + H(Z 21o Y))
(36)
= HZZe Zuy Y) / Zdy y),
As for AH2, it can be known that:
AH₂ = H (Zi Zie,1: Z HZ 1 Z Z
(H( Zag Zel, Y) + H (Zly Zei (2)
H (Ztt Zeg Y) + H H(Z 2 22(1)))
(37)
(dg Zot, Y) H (2 dy adyy Zot Y). HZ lo,t
By substituting Eq. (36) and Eq. (37) into Eq. (35), we obtain:
Al c,ti (Y) = AHI - All2
| (Z te,t1 Y) H (Z k2)
byy
y))
H (2 (Z Zzt, Y) 11 (2
(H Iee Zuy Y) HZZ Lee Zet, 8))= (Z Zuy Y) H
(Z
dy ie,
Zel Y)
term
term
(38)
From Eq. (38), since causal factors typically exhibit higher similarity to each other than to spurious factors, for a new given Z the entropy decay with respect to Zud is generally greater than the entropy decay with respect to z Therefore, it can be derived that
(
beti* ( I( x 2e (Y)
AI( a. 2e
Y) > 0,
(39)
Hence, dynamic causal factors zdy is the maximizer of Eq. (32). This completes the proof.
B.3.4. PROOF OF THE MAIN RESULT
Theorem 2.
Let (Xi), (X,) and Z denote the representations and drift factors at the time domain De, obtained by training the network through the optimization of Lsvve. Then the predictor constructed upon these components is the optimal causal predictor as defined in Def. 2.
Proof. As LSYNC Levolve + 01 CMi + Ceaeausal, according to Lem. and Lem. by minimizing losses LMI- Cemusal and the cross-entropy loss in Levalve, Tb (X,) and o (X,) will gradually approach the ground truth static causal factors Z and dynamic causal factors Zdy, which exclude any spurious factors. In addition, optimizing the loss Levolve make the drift factors | approach the drift of causal mechanisms. According to d-separation, We can conclude that the predictor built on top of these satisfies Y [Z.t 2u4y] (X,), d (X,). 2 Meanwhile, the mutual information between representations and the target can be maximized by minimizing the cross-entropy entropy loss in Levolve- Therefore, it can be concluded that the resulting predictor satisfies the criteria for the optimal causal predictor as defined in Def. This completes the proof. L.
18

---

Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
C. Implementation Details
C.1. Mini-batch Weighted Sampling
Mini-batch weighted sampling (MWS) strategy, proposed by (Chen ct al., 2018b), introduces a biased negative entropy estimator based on the principle of importance sampling. Although biased, this estimator straightforward to use for estimation and does require any additional hyperparameters. Suppose that the distribution of the dataset samples is p(n), the dataset size is N, and let BM {mi,'" (MM) be mini-hatch of M indices where each element is sampled i.i.d. from p(n). Let p( (BM) be the probability of any mini-batch By sampled from the dataset, then
y [log {()) = m),,,)) [log (=(()))
(x.,.) log E, (p()) (n) [n)]
(z. [log B(xx)
((=(n)
(40)
p(BM) p(BMr m))
M
2 Bn log BBsrrp(BBrrin In))
Σ [(((n)))))]
M Σ (<(()))))))
= Ex( Plg BBurr((BBrln)
NM ==1
therefore, By(=) log q(z) can be estimated using the following estimator:
1 M
M
E (2)) log (())
logl
NM Σ (<<(n)))n))]]-
(41)
M
C.2. Gumbel-Softtax Trick
To sample k-hot vector from an N-dimensional vector, we use the Gumbel-Softmax trick (Jang et al., 2017) and perform samplings without replacement. Specifically, let be score vector of dimension N, for the i-th (i [A]] sampling, using the Gumbel-SSottax trick, the sampled vector m is obtained by
exp((log 8f + +5)/T) N Σ exp((1og sty +
m
=-g(
log uz). log(
us Uniform(O, 1),
(42)
where J is the temperature hyperparameter. Following common setting (Chen al., 2018a), we set r = 0.5. After each sampling, we find the position of the largest element p argmaX,e[N] mi, and s(p) very small value acquire scores s*+1, then continue with the next sampling. This process continues for k samplings, and the sampled k-hot vector m = Glumbel-Softmxx(., k) is obtained by m, = maxie[k] m/.
C.3. Network Structure
For the static variational encoding network 9g., we utilize feature extractor to implement. The dynamic variational encoding network qe can be implemented using a feature extractor with the same architecture as but without sharing network parameters, followed by an LSTM network. The prior p(z =x))) is obtained by Fdy, using one-layer LSTM network. We implement 'e by dynamic inference network, which takes the one-hot code of as the input and output the categorical distribution. The prior p(=ç (= is acquired in manner similar way to p(=,"t dy
2 zdy), and is modeled by a prior netweotk Fd. In order to reconstruct the data æ1.7 and the labels yı.7, we utilize the appropriate decoder D and linear classifier W. To ensure fairness, the feature extractor and decoder in each task are used the same as LSSAE (Qin et al., 2022). The masker implemented by a 3-layer MLP which randomly initialized. During training stage, the sampling of 2, and x are performed using the reparameterization trick (Kingma & Welling, 2014), and x is acquired using the Gumbel-Softmax reparameterization trick (Jang et al., 2017).
19

---

Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
C.4. Model Inference
In this work, to make the prediction of æ; sampled from the target domain D., we need leverage the static causal representation &st the dynamic causal representation Φ and the drift factor Z simultaneously. Therefore, during inference stage, we use de to infer & and &et is obtained through the masker m I.e., & = m (f.'). For Z we employ the trained network of prior p(= |= (=2() to infer without requirement of 9g To get d we store the state variables of qu from the last training domain in the hidden state bank B at training time. Then, we randomly select a test batch from B and send to qo to infer / from 1 and 2 = m. ( ). At the same time, the state variables in B is updated to the state variables of the LSTM network in the current domain, in preparation for the prediction the next domain. Ultimately, hht c,
bd and Z are feed into the classifier W for prediction.
D. Algorithm of SYNC
The training and testing procedures of SYNC are shown in Algorithm 1 and 2.
Algorithm 1 Training procedure for SYNC
I: Input: sequential source labeled datasets S; static variational encoding network qei dynamic variational encoding networks qa. 4 and their corresponding prior networks Fd r'd decoder D and classifier W; masker m and mž hidden state bank B; total number of epoch N Nepoch and number of iterations per epoch Nu.
2: Randomly initialize all the networks.
3: for epoch 1, 2,2,... .Neooch do 6.
4: Set the hidden state bank
5: for it in 2.2..... ,Nir do
6: Sample training date (æ1.7, (e:r), from S1:7-
7: Assign 20 +- 0, 2 + 0. generate prior distributions p(=," (222) p( 12 1==x), 1,.., T via Fuly, F
8: Generate posterior distribution {x2²" (x)) via qt. generate posterior distribution (z da xx), 1,2, T and
hidden state str via qo. generate posterior distribution q(> >=< yr). t = 1,2, T via gç.
9:
Send 2 and 2 sampled from the posterior distributions of q(<]*|<z) [x,), 1, 2..., T and {( | xz),
1, 2, ..., T, along with the means of the posterior distributions pftr and p to the corresponding mask module.
> Calculate Levolve for 9v, qe, 9; Fdd Frd D, W, mi and may
11:
Calculate LMi for qu: qo, 9ç, Fdy Fd.
12:
Calculate Ceausal for m
st and m.y
13:
Update all modules by the summary of these loss.
14:
end for
15:
end for
Algorithm 2 Testing procedure for SYNC
1: Input: sequential target datasets T, static variational encoding network gu: dynamic variational encoding networks ga: prior network Fd; masker m and m classifier W; hidden state bank B.
2: Create new bank B².
3: for D, E T do
4: Set
5: for æ E D, do
6: Randomly sample hidden state st from the hidden state bank B.
7: Obtain u and st' by feeding and st into go. Store st' into B'.
8: Obtain 4* by feeding æ into go. Sample z via Fd
9: Send st uu to the corresponding mask module, and use the resulting representations along with 2d to make
prediction via W. 10: end for
11: Assign B +- B'
end for
20

---

Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
52
92.0
RMNIST
portraits
50
91.0
Farrals
30
48
90.0
20
46
89.0
88.0
10
10
44
87.0
42
86.0
0
10 15 20
0
10
15
20
0.4
0.7
0.8
85.0
0.4
0.5
0.6
0.7
0.8
iterations (x10²)
iterations 10P)
Mask Ratio
Mask Ratio
(a) Loss curve SYNNC-
(b) Study on mask ratio r.
Figure 7: (a) The loss curve of LSYNC on RMNIST and Portraits during the training phase. (b) Ablation study of SYNC to the mask ratio т on RMNIST and Portraits.
E. More Experimental Details
E.1. More Details on Datasets
Circle (Pesaranghader & Viktor, 2016) contains evolving 30 domains (15 source domains, validation domains, and 10 target domains), where the instance are sampled from 30 2D Gaussian distributions.
Sine (Pesaranghader & Viktor, 2016) includes 24 evolving domains (12 source domains, 4 validation domains, and 8 target domains), which is achieved by extending and rearranging the original dataset.
Rotated MNIST (RMNIST) (Ghifary al., 2015) is composed MNIST digits with various rotations. We follow the approach outlined in (Qin et al., 2022) and extend it to 19 evolving domains (10 source domains, validation domains, and 6 target domains) by applying the rotations with degree of {0", 15",
,5,,0 30 1880} in order each domain.
Portraits (Yao et al., 2022a) is real-word dataset consisting of photos of American high school seniors over 108 years (1905-2013) across 26 states. We divide it into 34 evolving domains (19 source domains, 5 validation domains, and 10 target domains) by a fixed internal over time.
Caltran (Hoffman et al., 2014) is surveillance dataset captured by fixed traffic cameras deployed at intersections. is split into 34 domains (19 source domains, validation domains, and target domains) by time predict the type of scene.
PowerSupply (Dau et al., 2019) is created by Italian electricity company and contains 30 evolving domains (15 source domains, 5 validation domains, and 10 target domains) for predicting the current power supply based on the hourly records.
ONP (Fernandes et al., 2015) dataset documents articles collected from the Mashable website within two years (2013-2015), aiming to predict the number shares in social networks. This datasct is divided into 24 domains (12 source domains, 4 validation domains, and target domains) according to month.
E.2. Training Details
Table 3: Training details on different datasets.
Dataset
B
Epochs
Optimizer
Learning Rate
Q1
Q:2
N
Circle
64
30
Adam
5c-6
0.02
0.6
20
Sine
64
50
Adam
le-5
1
0.001
0.6
32
RMNIST
48
120
Adam
5e-4
0.005
0.001
0.6
32
Portraits
24
100
Adam
5e-5
0.05
0.002
0.6
32
Caltran
24
100
Adam
le-5
0.005
0.002
0.6
32
PowerSupply
64
50
Adam
5e-6
0.001
0.01
0.6
32
ONP
64
50
Adam
5e-6
0.001
0.01
0.6
32
Training details different datasets are shown in Table 3, where B denotes the batch size, on and 02 represent the trade-off
hyper-parameter for the loss function LMi and Ceausal, respectively. т is the mask ratio of the masker and N represents the dimension of the latent space. All experiments in this work are performed on a single NVIDIA GeForce RTX 4090 GPU with 24GB memory, using the PyTorch packages. Following (Qin et al., 2022), the intermediate domains are utilized as validation set for model selection.
21

---

Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
3
2
2
1
1
0
2
4
-6
-
-4
(a) Original
(b) Gradual
(c) Abrupt
(d) Noise
Figure 8: Visualization of decision boundary, where the positive and negative classes are colored in red and blue. (a)-(d)
show the decision boundary of Circle-Orignal, Strcle-Gradual Circle-Abrupt and Circle-Noise, respectively.
E.3. Additional Results
E.3.1. COMPUTATIONAL COST
To evaluate the computational cost of our approach, we conduct experiments on the RMNIST and Portraits datasets, recording the memory cost and runtime per iteration. As shown Table and Table 5, it can be found that our method comparable among the considered methods. Specifically, the two indicators of our method are roughly similar to those of ISSAE and MMD-LSAF, with the memory cost lower than that of MMD--SAE and the runtime per iteration lower than that of GI. Moreover, it is worth noting that SYNC consistently outperforms all baseline methods, achieving an average accuracy improvement of at least 1.2% and the lowest accuracy improvement of 5.3% across all datasets. This suggests that only slight increase in computational cost is required to achieve performance improvements.
Table 4: Comparison of memory cost (GB) on different datasets.
Dataset LSSAE
MMD-LSAE
GI
SYNC
RMNIST 5.06
5.24
3.81
5.19
Portraits 17.01
17.20
14.48
17.14
Table 5: Comparison of runtime per iteration (s) during the training phase on different datasets.
Dataset RMNIST
LSSAE
MMD-LSAE
GI
SYNC
0.12
0.13
2.80
0.17
Portraits 0.35
0.36
7.84
0.42
E.3.2. CONVERGENCE ANALYSIS
In Fig. (a), we plot the loss curve of our method during training phase. From the results, it can be observed that the total loss of SYNC decreases steadily until convergence RMNIST and Portraits. This suggests that, although loss function comprises multiple components, the optimization path formed by their combination remains relatively smooth, with the components working synergistically to promote convergence. By optimizing LSYNC, the model can progressively capture the underlying evolving patterns the data distribution and learn time-aware causal representations, thereby mitigating the influence spurious correlations that may arise from data bias, ultimately enhancing the temporal generalization.
E.3.3. STUDY ON DIFFERENT MASK RATIO
We conduct experiment RMNSIT and Portraits explore the impact of mask ratio in Fig. 7 (b). It can be found that SYNC is little bit sensitive Based empirical results, mask ratio of 0.6 proves more effective for extracting causal factors, thereby enhancing generalization performance. Model performance improves when the proportion of true causal factors within the mixed factors aligns more closely with the selected mask ratio.
E.3.4. MORE DISTRIBUTION DRIFT TYPES
In order to further verify the robustness of our proposed method, we follow (Qin et al., 2022; 2023) and conduct experiments on three additional circle-base distribution drift datasets (Circle-Gradual Cirele-Abrupt and Circle-Noise) to demonstrate
22

---

Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
Table 6: The comparison of the accuracy (%) between SYNC and other baselines. "Wst" denotes the worst performance of the dataset on all test domains, and "Avg" denotes the average performance of the dataset on all test domains. The best result are marked in bold.
Gradual
Abrupt
Noise
Overall
Method
Wst
Avg
Wst
Avg
Wst
Avg
Wst
Avg
LSSAE
32.0
53.1
44.0
57.8
29.0
53.8
35.0
54.9
MMD-LSAE
34.4
65.5
45.0
68.7
32.0
53.8
37.1
62.7
SYNC
49.0
67.2
60.0
66.7
69.0
77.0
59.3
70.3
the effectiveness of SYNC in more complex scenarios. As shown in Fig. 8, the decision boundary of Circle-Gradual changes gradually, and decision boundary Circle-Abrupt changes suddenly at a certain moment, while the decision boundary of Circle-Noise fluctuates irregularly due to the influence of noise. These three drift types can simply simulate slow variations, drastic fluctuations, and irregular variations in real-world data distribution. We evaluate our method with LSSAE (Qin al., 2022) and MMD-LSAE (Qin al., 2023) on these synthetic datasets. As shown in Table 6, our method significantly outperforms the baselines in both metries, achieving at least 12.2% and 7.6% improvements in Wst and Avg average across all datasets, respectively. This suggests that SYNC has the potential to handle more complex and diverse drift types.
E.4. Full Experimental Results
We provide complete experimental results of SYNC and other baselines. It can be found that in most scenarios, approach achieve the state-of-the-art performance, showing the effectiveness of SYNC in non-stationary tasks. It is worth noting that for the experiment of SDE-EDG on PowerSupply, we report the results using the same backbone architecture as LSSAE to ensure fair comparison.
Table 7: Circle. We show the results each target domain by domain index.
ALGORITHM
23
26
29
Avg.
ERM
53.9 35
55.8
= 4,8
5,2
44.7 6.3
56.9 = 4.4
47.8
41.9 = 7.5
41.7 5.9
3.0
47.8
5.7
49.9
Mixup
3.
51.7
= 4,0
4..
43.6
16.9
47.8
58
41.9
41.9 7.5
41.7
5.9
54.2
47.8
5.7
484
MMD
50.0
4.
55.0
6.
608
49.7
41.9
41.7
54.2330
47.8
50.7
MLDG
57.
57.7
55.119
46.4
56,9444
478
41.9
41.7
54.233
47.8
50.8
RSC
45.
51.4
49.4
43.6
56.9444
47.
41.9
41.7
542230
4.7
48.0
MTL
61
6.
53.31:511
6..3
49.2
43.3
45.8
54.215.77
42.2
51.2
Fish
51.7
53.1
49.44:55
43.6
569
478
41.9
41.7
542233
47.
48.8
CORAL
6.2
6.49
60.0
56.4
4.
478
41.9
41.7559
54,233
4.7
53.9
AndMask
42.8+34
5..6+4.11
494445
43.6
58
56,9
4.4 47.
41.9+55
41.7+599
$42+30
49.7
47.9
DIVA
4.5
76.3
4..3
74.7
5..7
67.0
6.1 5.
62.0556
6..3+
70,3
620
67.9
IMM
57.8
59.4554
569949
48.
4,
478
41,9
41.7
542.30
478
51.3
IB
7..0
68.044.1
58.0455
44.0
55.0
49.0
42.044.1
45.0
55.033
44.0
53.9
DDAG
52.0+34
53.043.7
51.044.1
44.0
55.0
45.0+42
49.0
42.0
45.0
550034
44.0
49.0
72.0+49
62.0
61.0+
58.0
49
56.0 + 4.7 4..0+47
42.0+22
4..0
550+40
44.0
54.4
LSSAE
9.
95.6211
93.5229
98.3
888
743436
51.9
52.11.11
465
48.455
73.8
DDA
7.00
56.0
51.0
44.0
55.0 0,7 49.0221
42.0
4..0
55.035
44.0
51.2
DRAIN
48.0
5270
54.0
47.0 14
58.0
520+35
45.0
4.90
580:49
47.0
50.7
MMD-LSAE
93.0+21
89.0
83.0
84.0
840
+ 4.4 880
79.0
76.0664
54.0
65.0
79.5
CTOT
1.
8.
19.0
77.
79.0112
720012
65.033.0
550009
$4.0
75.2
SDE-EDG
9.70
99.0
9.
94.0
95.0
93.0788
74.0
6.00
45.0
56.0
81.5
SYNC
87.0 95.0
87.0
94.0
88.032
84.0333
85.0
80.0
6.00
80.0
84.7
23

---

Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
Table Sine.
We show the results each target domain by domain index.
ALGORITIM
17
18
19
20
21
22
23
24
Avg.
ERM
71.4 6.1
91.0
1.5
81.6
2,4
53.4 2.9
51.1
6.7
54.3
4.7
49.5 1 4,8
51.7
5.0
63.0
MIXUP
63.1 =
93.5
1.7
80.6
128:299
60.3
54.2 2.7
49.5 4,4
49.3
62.9
MMD
57.0
47.1
5
MLDG
6.2
67.7
52.1=54
50.7
1.8
5..1
5.73
49.5
51.7
63.2
RSC
6.6
1,9
52.8 84.5226
5..1
44,4
49.5
51.7
61.5
MTL
70.6=66
91.6
79.9334
5,.0
60.3
53.6
4.3
46.9
Fish
2.7
5.7
4.73
49
5.07
62.3
CORAL
60,0 5.1
4..1
48.6
50.7
49.7 6.2
48.6 4.6
4.0
5,07
51.6
AndMASK
42.9
542270
71.9 1.9
864
2.
88.1
7.74
69.3
DIVA
79.0
60 19
47.6=26
50.0
51,9
38,6
5,0
52.9
TRM
66.9
31
88
6.
57.2
5.8
5.17
2.
5.
63.2
IIB
56.0
2.21
1.
4.0
55.1 4.9
54
50,7 39
61.3
IDAG
86.0 3.2
4-,0
49.--37
50.0 38
55.1
4.5
5.1
49.6
3.8
50.0 3.3
57.1
-.7
4.
89.1
00
60
6.
49.8
2.
65.2
LSSAE
93.0 1.7
86.9
69.2 1.5
1.
68.8
2.5
1.88
63.9
3.1
714
DDA
0.7
4..2
74.2 1.6
1.
75.5
1.1
70.0
59.1 111-24
74.0
66.6
DRAIN
43.0=
4,76
09
89. 1.1
828 4.8
78.5 25
70,2
4,1
92.0
71.3
3.2
43,9
79.7
0.7
98.8
2.3
714
CTOT
43.2 1.9
52.6
62.1
0.9
87.4
78.4
28
71.6
77.4
1.3
2.
67.
SDE EDG
99.0 0.7
96,9 0.66
90.0
4.2
88.8
0.6
64.9
2.2
52.4
0.3
43.0
1
4.9
42.3
4.0
72.2
SYNC
67.0 4.0
89.0
84.5
322
79.8
75.5
28
68.9
11.
60.0
2.7
84.0
3.1
76.0
Table 9: RMNIST. We show the results on each target domain denoted by rotation angle.
ALGORITHM
130
140
150"
160
170'
180
Avg.
ERM
568 - 0.9
44,2 0.8
37.8
0.6
38.3 0.8
40.9 + 0.8
43.6
0.8
43.6
MIXUP
61.3=
47.4
39.1
0.7
38.3 0.7
40.5
0.8
42.
0.9
44.9
MMD
59.2
46.0
0. 0.7
0.8
41.6 0.7
43.7
08
44.8
MLDG
$7.4 0.7
44,5 0.9
08
37,5 0.8
39.9 0.
42.0
= 0.9
43.1
RSC
39.8
41.6
41.7
MTL.
54.8 0.9
43.1 +.08
364+008
36.1 +
39.1 0.9
40.9 008
41.7
FISH
60.8 0.8
0.
0.
0.7
39.0
40.7 0077=07
44.2
CORAL
58.8
46.2
38.5++00
41.3 11.3++05
43.5
44.5
ANDMASK
$3.5 0.9
4.
37.8
38.6 + 0.8
40.8 + 0.8
43. 0.8
42.8
37.6
36.9
38.1
40.1
42.7
IRM
47.7=09 0.9
38.5 0.7
0.71
35,7 0.8
37.8 +
40,3 = 0.8
39.0
594 = 0.7
49.5 4 0.8
42.4
38.5 0.8
39.6
40.
088
45.0
IDAG
56.7
0.8
0.
0.8 0.7
44.1
GI
61.6 0.9
46.4 0.9
40.0 38.2
40.1 10.1:008
40.1
44.6
LSSAE
40.3
DDA
60.7 0.8
50.0
42.6=
0..6
38.0
39.7
45.1
DRAIN
59.5 088
+8.4
0.7
37.2
+9.6
4
43.8
MMD-LSAE
03
0..6
43.1
4.
48
49.2
CTOT
68.2 0.7
4.
36.7
32.3
31.7 088
44.8
SDE-EDG
75.1 09
49
0.88
0.77
09
52.6
SYNC
65.1 0.7
02
4.1
4.3 4.9
46.6 09
50.8
Table 10: Portraits. We show the results on each target domain by domain index.
Avg.
ERM
75.5 0.9
838 1 0.9
88.5 0.8
93.3 1 0.7
93.1 1: 0.6
92.1
1 0.7
90.6
0.8
$1.3
88.5 0.9
87.9 1:
$7.8
MIXUP
0.
838
09
08
93.3
93.4
92.1
: 0.7
90.6
0.8
84.3
85.5
87.9
87,8
MMD
1.00
83
9.70 08
91.9 93
90.9 0.7
$1.7
1.4
0.
85.8
67.3
MLDG
7..8
90.1
94.3 94.3
9.
92.0 0.7
90.8
85.6
1.1
0.8
87.6
1.6
88.5
RSC
1.9
0..7
0.7
9.73
9.77
9..0
90.0 0.7
$4.6
0.8
85.8 1
87.3
MTL
1..2
86.5 0.51:000 0.8
90.9
94.2
93.8
92.0
91.2 0.7
86.0 = 1.2
87.4 +
$9.0
Fish
86.9
0..
93.3
+
86.2
+.9
N7.7
CORAL
74.6
84.6 4
8.
93.3 4
92.7 1 0.7
91.5 1 0.7
90.7
0.7
84.6
= 1.5
88.1 0.9
85.9 1: 1.9
$7.4
AndMASK
6.1
703 +
67.0
70.2
74.1
64.7
77.3+11.1
74.9
DIVA
7.22
86.6 0.8
88.8 0
93.5 4 0.7
93.1 0.6
91.6 0.6
91.1
0.7
84.7
= 1.3
1..1
87.0 4:
88.2
IMM
X3.5 04
91.0
90.4 0.7
87.3
M0A
86.7
K5.1
85.4
IB
1.
1.1
91.8 1.2
95.8 + 0.8
94.9 1 0.7
92.2 1.4
91.7
1.2
87.
= 1.1
89.3 1
88
1: 1.5
89.7
DAAG
1.1
86.2
91
94.0
89.3
K7A
88.4
GI
86.6 1..2:12
90.8 1.1
95.3 +
93.1 1 1.2
89.3 1.1
889
1.2
84.1 188
87.7 1.0
87.5
4: 2.0
88.1
ISSAF
0..7+009
87.1 0.8
9.7
94.3
94.3+606
91.2-01 0.7
99.6+00.8
86.9
DDA
7600
88.6
93.6
92
92.9 066
90.3
08
84.3 1.2
88.7 0.8
85
1.2
87.9
DRAIN
92.
87.9
MMD-LSAE
809 0:0.9
08.6
0.
9.
9.99
91.0 0.9
92
88.1
1.2
004
89.7 4: 1.1
90.4
90.1106
87
$3.9
SDE-EDG
78.6 0.8
86.6 09
90.1
9488
94,5
93
9211
879
1.3
89.6 0.9
89.0 + 1.1
89.6
96.0 92
0.7
95.7
$7.8
9.9
24

---

Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains
Table 11: Caltran. We show the results on each target domain by domain index.
ALGORITIM
29
Avg.
ERM
29.9 : 3.5
88.4 2.1
61.1 +
$6.3 +
90.0 1.6
60.1 +
55.5 =
88.8 = 24
57.1 3.5
50.5
= 5.2
66.3
MIXUP
1.9
89.0 2.0
61.8 1
55.7 2.9
88.2
58.6
$2.3 3.
88.6
57.1
55.1
:
66.0
MMD
2.1
92.7 0
$6.4
39.1
93.6 1.7
42.8 42.
92.1
42.1 3.88
29.4
1: 3.8
57.1
MLDGG
54,8 4.1
88.6
2.6
62.2 + 3.6
55.1 4.1
88.3+17
00.443
51.7
89.0+19 19
56.5
55.3 + 4,8
66.2
RSC
3.0
2.6
24
3.
2.
56.0
67.0
MTL
4..0
87.2 + 2.5
64.9 + 3.9
60.0 + 4.8
2.2
00.6 3.5
52.6 0 3.7
43.9
58.2
65.7 5.6
68.2
Fish
61.1 3.5
88.2
+ 1.5
64.7
3.1
22
575
87
3.7
63.0 6.1
68.6
CORAL
3.0
90.8
2.0 2.0
6.8
2.5
92.0 1.7
36
90.9 5..0
50,9 2.48
65.7
AndMASK
2.2
92.7
1.7
56.2 1. 3.8
39.1 3.2
93.6
1.7
$1.6
42.6 22 2.9
92.1 2.2
3.7
29.9 1:
56.9
DIVA
60.6::24.
90.1
1.7
67.5 4:
58.9
88.4
4 2.8
58.7
3.3
53.8
4.8
62.0 3,4
69.2
IKM
4.7
90.8
1.7
608 +
$2.9 + 3.1
91.8
1.7
56.6 3.1
52.1 +
90.9 2.6
55.6 3.9
43.1 +
64.1
6.12
3.0
3.7
68.1
IDAG
3.8
91.2 2: 1.2
68.5 +: 3.2
36.2 2.8
88.8 1.9
6.1
53.9
89.1
69.4
$3.5 - 4,1
69.7
1.9
53.1
73.0
70.7
LSSAE
3.
92.1
+ 2.0
62.6
0664443
58.8 4.4
92.9 1.6
6.
3..
2.2
6.8
67.4 4
70.6
DDA
3.40
92.6
4 1.8
56.8
1: 0.9
59.0 2.8
4
2.
$2.9 2.3
92.9 2.2
5.3
62.9 5.8
66.1
DRAIN
3.3
83.8 + 1.0
65.7
+ 2.2
628
77.9
3.7
55.7 3.7
7
60.6+388
75.7 75.7+550
69.0
MMD-LSAE
3.77
17.4
65.7
+
60.4
27
0.4
36.9
85
75.0 2.
69.6
1.
33
57.0
71.0 2.
66.9
SDE EDG
70.5 2.9
88.8
4.9
66.1 1 288
55.1 1 2.6
39.5
58.6 2: 3.3
88.2 3.4
68.9 3.5
72.2 1
71.3
SYNC
2.6
90.7
1.3
71.1
59.4
93
63.5
65.0
62.6 4.5
72.2
Table 12: PowerSupply. We show the results on each target domain by domain index.
ALGORITHM
21
22
23
24
25
26
27
28
29
30
Avg.
ERM
69.8 1.4
70.0 + 1.4
69.2 + 1.3
64.4 + 1.5
85.8
1.0
76.0 1.
70.1 = 1.5
69.8 = 1.5
69.0
1.3
65.5 + 1.5
71.0
MIXUP
69.5
68.3
64.3
70.1
69.2
1.6
70,8
MMD
70.0 1 1.3
69.7 1 1.4
68.7 + 1.4
64.8
1.5
85.6 1.0
76.1
70.0 = 1.5
69.5 1.4
68.7 2 1.3
65.6 15.6:10 1.5
70.9
MLDG
68.6
64.6
70.1
69
65.6
70,8
RSC
69.9+ 1.4
69.6 1.4
68.6
64.4 1.5
86.6+10 1.0
76.3++13
70.0 1.5
04+14 1.4
684 1.3
68.411 1.5
70,9
MTL
69.6 1A
69.4
68.2 1.6
64.2 1.5
874
76.6
69.9
1.
1.5
64.6
70.7
Fish
1.4
69.1 1.4
68.2 14
64.2
87.3 1.0
76.6
1.9
6..2
1.3
65.2
70.8
CORAL
69.9 1:11 1.4
69.7 1.
68.9
64.6
1.4
86.1 4 1.0
76.3 4 13
70.0
69.5
1.5
65.7 L 1.5
71.0
ANDMASK
69.9
+.
68.2
64.0
87.4009 0.9
76.7
70.0
6..1
6..0 1.3
64.7 + 1.5
70.7
DIVA
1.
1.3
68.2 14
63.9 1.5
87,5 1.0
76.5 13
69.9 1.5
1.5
1.
64.7 1 1.5
70.7
IRM
1.
69.5
68.3
87.2
76.5
70,0
65.0
70.8
IBB
1.3
69.5 1.4
68.2
64.5 14
86.9 1.1
764 1.2
69.9 = 1.4
69.0
68.1
65.1 1.4
70.8
IDGG
70.7
70.4
1.4
70.0 1.2
66.1 1.3
83.5
1.1
76.3 1.1
70
1.4
1.
6..9
66.7
71.2
GI
1.4
71,0
1.4
70.5 1 1.5
69.6 1.5
80.7 2 1.1
68.4 13
72.9
72.0 1.3
1.3
66.5
71.4
LSSAE
70.0 1.4
69.8
1.4
69.0 +
65.4
14
85.1
1.1
76.0 + 14
70.1 + 1.7
69.9
1.3
69.0 1.6
66.3 1.4
71.1
DDA
69.8
72.4
70.5
638
73.1
7.
1.7
70,9
DRAIN
1.3
1.0
69.3 1.1
65.5 1.5
83.6 1.0
75.8
70,3 = 1.3
1.
6..9
66.4
71.0
MMD-LSAE
14
718
65.2114
70.7
14
1.
714
CTOT
70.3
71.4
73.7
4 12
70.4 1.2
72.0 12
63.6
75.2
11
72.8 1.2
70.6 1.2
71.1
SDE-EDG
1.1
69.5
1.3
68.2
1 1.1
64.1 12
87 111
76
70,0
1.
68.0
65.0
7088
SYNC
1.3
72. 1.2
71.7 4
66.9
80.0
1.
70.6
1.2
72.
71.1
1.2
67.3
67.31.
1.
71.7
Table 13: ONP. We show the results each target domain by domain index.
ALGORITHM
17
18
19
20
21
23
24
Avg.
66,6
1 1.2
67.0 1
64.6
65.9
Mixup
67.0
1.2
67.3 1 1.1
66.1 + 1.1
67.2 1.1
66.7 1 11
64.1 1 111
64.5
1.0
65.0
1.1
66.0
MMD
58,9 1.1
52.5 +
54.6
48.1
47.1
49.2+1.1
$1.3
MLDG
67.0
67.0
66.0
66.3 1.1
668
63.9
6..0
6
65,9
RSC
64.9 1.2
65.2 122
65.0
66.0 111
65.5
626
6..7
6.1
64.7
MTL
1.1
66.5
65.7
1.
1.:5100
63410
1.
1.2
65.6
Fish
67.2
66.8
+ 1.2
65.7
1.1
12
67.3
6.22
6..1
6.17
65.9
66.9
65.9
66.6
1.
1.
AndMask
59.5 1.2
56.1 + 1.2
$6.4 + 1.I
$6.0 12
1.2
$1.9 1.1
$..0
51.
$4.6
DIVA
67.8
67.2 1.3
66.411
6.
67.31111
63
650+1.11
6.2
66.0
IRM
6..1
65.1 +
64.7
65.2
6.
61.9
1.2
6.1
64,5
IBB
1.2
67.5
66.8 1 1.1
67.3
1.
632
1.2
1.2
1.11
66.3
IDAG
1.2
67. 1.2
67,8
66.2112
1.
638
1.3
1.2
66.4
GI
67.6+ 1.2
66.9 1.1
66.3 1.I
66.4 13
67.1
64.+12
1.2
649 12
63.6
65.9
LSSAB
66.2 1.3
6.
64.510
14
1.1
66.0
DDA
67.7
66.2 1.1
66.6 1.2
66.3
1.
6.
1.
6.
65.8
60.7
59.8
61.2
61.6112
606
11
1.1
65.8
MMD-LSAE
61.8
66.1 1.3
66.7
67
67
61
1.
65.4 1.2
66.4
CTOT
62.7 1.4
66.3 1
65.5 18
16
63.9111
632112
65.3
6..
64.3
SDE-EDG
67.8 +: 1.2
66.8 + 0.8
65.6
6..7000
66.0
6..1
1.
646
65.6
SYNC
66.4 +: 1.2
66.0
65.0 1.1
65.9 1.1
66.9 1.0
64. 1.2
64. 1.1
64.6
1.2
65.6
25


Reviews:
[{"review_id": "tzJfGNZifE", "reviewer": "Reviewer_GuDA", "summary": "This paper proposes SYNC, a method for improving temporal generalization in evolving domains by explicitly disentangling static and dynamic causal representations. It introduces a sequential variational autoencoder (VAE) with mutual information minimization constraints to separate static and dynamic causal factors. Experimental evaluations on synthetic and real-world datasets show that SYNC achieves better generalization performance than existing causal and non-causal domain generalization methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. The contribution of causal factors to performance.\n\nR1. Thanks. We perform a further analysis of causal factors. Please refer to R2 of Reviewer RWjv for details.\n\n> Q2. Model assumptions and their impact on performance.\n\nR2. Thanks. Given the challenging of the EDG problem, most existing methods make appropriate assumptions to derive their objectives. To name a few, LSSAE [1] assumes the latent variable's Markov property, while DDA [2] assumes consistent evolution across domains. Our approach adopts a causal perspective, relying only on the fundamental SCM assumptions like Markov condition, to build a reasonable SCM for modeling time causal factors. These widely used assumptions have been validated in real-world scenarios in literature [3-5]. Besides, our method generally outperforms other methods on real-world datasets in our paper, supporting the assumptions.\n\nSince causal sufficiency and faithfulness are relatively important in our assumptions (violating causal sufficiency disrupts static-dynamic independence, while violating faithfulness affects conditional dependencies), we conduct experiments to evaluate their impact. Results below show little performance change. Moreover, incorporating these relaxed assumptions improves performance.\n\nFinally, we conducted experiments on the more challenging FMoW. The results below show that SYNC outperforms others, demonstrating its applicability in more complex scenarios.\n\n||RMNIST|Portraits\n|-|-|-\n|SYNC|49.6|89.9\n|SYNC w/o sufficiency|48.9|89.2\n|SYNC w/o faithfulness|50.8|90.8\n\n||LSSAE|SDE-EDG|SYNC\n|-|-|-|-\n|FMoW|42.8|44.2|46.6  \n\n[1] LSSAE, ICML’22.\n\n[2] DDA, AAAI’23.\n\n[3] MatchDG, ICML’21.\n\n[4] CIRL, CVPR’22.\n\n[5] iDAG, ICCV’23.\n\n> Q3. Comparison with transformer-based DG methods.\n\nR3. Thanks. We discuss transformer-based DG methods here. [5] generates data from unseen domains using CycleGAN, while [6] investigates the deployment of vision transformers (ViT) in DA and DG. However, neither addresses the DG using transformers. Here we compare SYNC with two highly cited transformer-based DG methods, i.e., DoPrompt [5] and GMoE [6]. While ViT helps capture robust semantic features, they fail to consider the continuous structure in EDG. The results below show SYNC's superior temporal generalization.\n\n||GMoE|DoPrompt|SYNC\n|-|-|-|-\n|Portraits|87.9|88.2|90.8\n|Caltran|70.1|70.4|72.2\n\n[5] DoTra, IJCNN’22.\n\n[6] Vision...Generalization, Neural Comput Appl’24.\n\n[7] DoPrompt, arXiv.\n\n[8] GMoE, ICLR’23.\n\n> Q4. Evaluation on more real-world benchmarks.\n\nR4. Thanks. We evaluate our method and baselines on FMoW and the results are detailed in R2. It is known that SYNC remains effective in generalizing in more challenging scenarios, exhibits the potential for deployment in complex real-world applications.\n\n> Q5. Computational complexity and scalability analysis.\n\nR5. Thanks. Our network largely follows LSSAE [1] and MMD-LSAE [9], with the only addition being two MLP-based maskers. Similar to them, the time complexity and memory complexity of sequential VAE are $\\mathcal{O}(T\\cdot B\\cdot [\\sum\\_{l=1}^LH^{l}W^{l}C^{l-1}C^l(K^l)^2+D^2])$ and $\\mathcal{O}(T\\cdot B\\cdot [\\sum\\_{l=1}^LH^lW^lC^l+D])$ respectively, where $T$ is the number of time domains, $B$ represents batch size, $D$ is the dimension of latent features. For the \n$l$-th layer of the decoder, $H^l$ and $W^l$ denote the output feature map size, $C^l$ represents the output channel, and $K^l$ denotes the kernel size. For the MI loss function, the time complexity and memory complexity are $\\mathcal{O}(T\\cdot B^2\\cdot (D+1))$ and $\\mathcal{O}(T\\cdot B\\cdot D)$, respectively. In the implementation, $D$ and $K$ are set to a relatively small value ($D=32, K=5$) and the complexity is acceptable.\n\nBesides, we conduct an experiment on FMoW using DenseNet-121. The results show the effectiveness of our approach in challenging real-world scenarios. Additionally, we record memory usage and runtime per iteration. The results below show that our method requires almost the same cost to achieve better performance.\n\n||LSSAE|MMD-LSAE|GI|SYNC\n|-|-|-|-|-\n|FMoW|35.2G/0.79s|35.3G/0.80s|28.6G/12.6s|35.3G/0.87s\n\n[9] MMD-LSAE, TPAMI'23.\n> Q6. Comparison with Koodos [10].\n\nR6. Thanks. Koodos considers the continuous temporal DG and models the evolving pattern via Koopman theory. Our method is tailored to the EDG and enhance temporal generalization by tackling spurious correlations through time-aware causal representations. We evaluate Koodos on two datasets and the results below verify the effectiveness of SYNC.\n\n||Circle|RMNIST|\n|-|-|-\n|Koodos|81.4|44.6\n|SYNC|84.7|50.8\n\n[10] Koodos, NeurIPS’24.\n\n> Q7. Practical robustness checks.\n\nR7. Thanks. We conduct a sensitivity analysis experiment on $\\alpha\\_1$ and $\\alpha\\_2$, the results are shown in Fig. 1 of https://anonymous.4open.science/r/2150-1CBD, indicating insensitivity within a certain range. Additionally, our evaluation on FMoW shows the method's adaptability to more complex scenarios.", "claims_and_evidence": "Claim: *SYNC achieves improved temporal generalization by modeling dynamic causal factors and causal mechanism drifts.*\n\nThis is partially supported. The experiments indeed show improvement over various baseline methods. However, results lack deeper analyses into why and how each causal factor contributes to performance.\n\nClaim of optimal causal predictor: Theoretically argued, but practically the verification is incomplete and lacks rigorous validation.", "methods_and_evaluation_criteria": "1. The current evaluation is primarily accuracy-based. Given the goal is generalization in evolving domains, metrics reflecting robustness (such as performance variance across evolving domains or robustness metrics under shifts) should also be considered.\n\n2. Good coverage of existing EDG and DG methods, but lacks a detailed comparison with e.g., transformer-based methods [1,2].\n\n[1] Domain Transformer: Predicting Samples of Unseen, Future Domains (2021)\n\n[2] Vision Transformers in Domain Adaptation and Domain Generalization (2024)", "theoretical_claims": "In this paper, the author presents two theoretical claims. Optimality of causal predictor for each domain is theoretically stated, but practical relevance and conditions for this optimality (e.g., assumptions like SCM correctness, Markov condition, faithfulness) remain unvalidated empirically. How sensitive is the method to violations of SCM assumptions?", "experimental_designs_or_analyses": "1. Datasets selected (Circle, Sine, RMNIST, Caltran) are standard but somewhat simplistic or dated. How representative are these of realistic, high-dimensional, complex evolving domain generalization scenarios? For instance, how does the proposed method perform on WILDS benchmarks from Stanford (https://wilds.stanford.edu/datasets/#fmow).\n\n2. Limited exploration of scalability (the proposed method's overhead, complexity, and memory cost are largely omitted). There is only one relatively limited comparison of computational cost in the appendix currently.", "relation_to_broader_scientific_literature": "The paper relates well to domain generalization and causal representation learning literature, clearly distinguishing itself from static causal DG methods.", "essential_references_not_discussed": "\"*Continuous Temporal Domain Generalization.*\" Zekun Cai, Guangji Bai, Renhe Jiang, Xuan Song, Liang Zhao. NeurIPS 2024.", "other_strengths_and_weaknesses": "S1. Clear methodological motivation, novel integration of causal modeling into evolving domains.\n\nS2. The theoretical analyses round up a rigorous paper.\n\nS2. Promising empirical performance improvements.\n\nW1. Over-complex method without sufficient empirical or theoretical justification for complexity.\n\nW2. Insufficient practical robustness checks (e.g., sensitivity to hyperparameters, data complexity).\n\nW3. Limited clarity and depth in experimental analysis, especially regarding scalability and real-world application feasibility.", "other_comments_or_suggestions": "Please refer to my comments above. My final score will depend on other reviewers' comments and rebuttal discussion, and I will adjust my score accordingly.", "questions_for_authors": "1. How robust is SYNC to violations of SCM assumptions, specifically regarding causal factor disentanglement and mechanism drifts?\n\n2. What is the computational complexity of SYNC, particularly the scalability of sequential VAE and MI computations?\n\n3. Could you also compare with the transformer-based DG methods given their relevance?\n\n4. Could you empirically evaluate the proposed method and other methods on more real-world benchmarks such as WILDS?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8rKGawt3Kd", "reviewer": "Reviewer_RWjv", "summary": "This paper proposes a framework called Static-DYNamic Causal Representation Learning (SYNC) to deal with distributional drift in dynamic environments for generalization. By designing a time-aware Structural Causal Model (SCM), SYNC models dynamic causal factors and causal mechanism drifts, leveraging a sequential Variational Autoencoder (VAE) framework combined with information-theoretic objectives to learn time-aware causal representations. Theoretical analysis demonstrates that this method effectively mitigates spurious correlations and learns the optimal causal predictors for each time domain. Experimental results on synthetic and real-world datasets validate its superiority and broad applicability in dynamic, non-stationary environments.\n\n## update after rebuttal\nAll my concerns have now been addressed. As previously suggested, I hope the authors can thoroughly revise the manuscript by adding a detailed description of the ablation study and carefully correcting the typos.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. The significance of Proposition 3.3 and its connection to SCM.\n\nR1. Thanks. Here we provide a detailed explanation of Proposition 3.3.\n\n**The Significance for learning causal representation:** The two points of Proposition 3.3 respectively guide the learning of static and dynamic causal representations. \n\nFrom Proposition 3.3 (i), it can be found that for various static factors in two consecutive temporal domains, the static causal factors $Z_c^{st}$ are those that maximize the conditional mutual information (CMI) under a given category. Since we extract static causal factors using network $\\Phi_c^{st}$, static causal representations can be learned by maximizing the CMI $I(\\Phi_c^{st}(X_t);\\Phi_c^{st}(X_{t-1})|Y)$. Proposition 3.3 (ii) shows that among various dynamic factors within the same temporal domain, the dynamic causal factors $Z_c^{dy}$ are those that maximize the CMI with the static causal factors under a given category. Therefore, static causal factors $Z_c^{st}$ can serve as anchors to facilitate the learning of dynamic causal factors $Z_c^{dy}$ by maximizing $I(\\Phi_c^{dy}(X_t);Z_{c,t}^{st}|Y)$.\n\n**Connection towards the SCM:** Although Proposition 3.3 can be derived straightforwardly using information theory, its inspiration actually comes from SCM and can be intuitively explained within the SCM framework. Since the two points of Proposition 3.3 are quite similar, for the sake of simplicity, we will explain using the second point. As shown in the time-aware SCM in Figure. 2, there is a clear collider structure and a fork structure, namely $Z_c^{st}\\rightarrow Y \\leftarrow Z_c^{dy}$ and $Z_s^{dy} \\leftarrow L \\rightarrow Z_c^{dy}$. According to d-separation, when conditioned on $Y$, both $Z_c^{dy}$ and $Z_s^{dy}$ are related to $Z_c^{st}$. Consider a boundary case where $Z_c^{dy}$ and $Z_s^{dy}$ are independent, this implies that the backdoor path between $Z_c^{dy}$ and $Z_s^{dy}$ is blocked, leading to independence between $Z_c^{st}$ and $Z_s^{dy}$, while $Z_c^{st}$ remains related to $Z_c^{dy}$. Proposition 3.3 generalizes this observation, under certain entropy inequalities, static causal factors are more strongly related to dynamic causal factors than dynamic spurious factors.\n\n> Q2. More details about ablation study.\n\nR2. Thanks. Here we explain the ablation study in detail and illustrate the contribution of the modeled causal factors to performance. As the table shown below, Variant A serves as the base method trained solely with evolving pattern loss $\\mathcal{L}\\_{\\text{evolve}}$. Variant B builds upon the base method by additionally training with MI loss $\\mathcal{L}\\_{\\text{MI}}$, serving as an ablation for $\\mathcal{L}\\_{\\text{causal}}$. Variants C and D build upon Variant B by incorporating additional training with static causal loss $\\mathcal{L}\\_{\\text{stc}}$ and dynamic causal loss $\\mathcal{L}\\_{\\text{dyc}}$, respectively. We conducted additional ablation experiments on the Portraits dataset, excluding RMNIST, and recorded their worst-case performance (W) and average performance (A). The results are presented below. \n\nFirst, both Variant C and Variant D show performance improvements over Variant B, validating the effectiveness of modeling static and dynamic causal factors. \n\nThen, it is clear that Variant C achieves a greater improvement in worst-case performance compared to Variant D, indicating that static causal factors can ensure stable generalization under continuous distribution shifts. However, focusing solely on them ignores evolving pattern in EDG, limiting further generalization gains. Learning dynamic causal factors captures features related to task evolving over time, enabling to generalize better to the current distribution. Variant D outperforms Variant C in average performance and provides evidence for this claim. \n\nFinally, SYNC learns static and dynamic causal representations jointly, achieving the best performance, demonstrating their significant contribution to overall performance.\n\n||| RMNIST (W/A) | Portraits (W/A) | Overall (W/A) |\n|-|-|-|-|-\n|Variant A | base | 40.5/44.1 | 78.1/89.2 | 59.3/66.7 |\n|Variant B | base+$\\mathcal{L}\\_{\\text{MI}}$| 41.9/45.7 | 78.3/89.4 | 60.1/67.5 |\n|Variant C | base+$\\mathcal{L}\\_{\\text{MI}}$+$\\mathcal{L}\\_{\\text{stc}}$ | 44.1/48.7 | 79.8/89.9 | 62.0/69.3 |\n| Variant D | base+$\\mathcal{L}\\_{\\text{MI}}$+$\\mathcal{L}\\_{\\text{dyc}}$ | 42.9/49.3 | 79.1/90.4 | 61.0/69.8 |\n| SYNC | base+$\\mathcal{L}\\_{\\text{MI}}$+$\\mathcal{L}\\_{\\text{stc}}$+$\\mathcal{L}\\_{\\text{dyc}}$ | 45.8/50.8 | 81.0/90.8 | 63.4/70.8 |\n> Q3. Typographical errors.\n\nR3. Thanks. As your suggestion, we will thoroughly revise the manuscript to improve clarity and readability.", "claims_and_evidence": "N/A", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "No, just in AI research community.", "essential_references_not_discussed": "Yes", "other_strengths_and_weaknesses": "**Strength:**\n\n1. The framework is conceptually solid and interesting. The introduced SCM module integrates both static and dynamic causal factors, expanding causal representation learning to dynamic, non-stationary environments.\n2. This paper provides rigorous theoretical analysis, proving that SYNC can learn optimal causal predictors for each time domain and mitigate spurious correlations effectively.\n3. Extensive experiments have been done on various synthetic and real-world datasets and the proposed method exhibits excellent generalization performance under various types of domain shifts.\n\n**Weakness:**\n\n1. The significance of Proposition 3.3 in learning dynamic causal representations is unclear. While (i) and (ii) seem reasonable here, what is the their connection towards the SCM?\n2. Has an ablation study been conducted on the loss term $\\mathcal{L}_{causal}$? What is detailed implementation of Variants A-D? I cannot find them in the main paper and appendix.\n3. The manuscript contains numerous typographical errors, thus I recommend the author to revise their manuscript thorough to improve clarity and readability.", "other_comments_or_suggestions": "- In Line 189, left half of the page. I guess X and Y should be $X := f^x(Z^{st}_c,Z^{st}_s,Z^{dy}_c,Z^{dy}_s, \\epsilon_x)$ $Y := f^y(Z^{st}_c,Z^{dy}_c,Z^d, \\epsilon_y)$\n- Lines 253-257, right half of the page. The maker should be $m^{st}_c$ and $\\Phi^{st}_c = m^{st}_c \\circ q^{ext}\\_{\\psi}$", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qejUA5tD56", "reviewer": "Reviewer_LQYB", "summary": "This paper addresses the challenge of generalizing deep models in evolving domains, where data distributions shift dynamically over time. The author claims that exiting Evolving Domain Generalization (EDG) approaches suffer from spurious correlations, which degrade their generalization ability. To mitigate this, the authors propose a time-aware Structural Causal Model (SCM) that explicitly models dynamic causal factors and causal mechanism drifts. They further introduce Static-DYNamic Causal Representation Learning (SYNC), an approach that integrates a sequential Variational Autoencoder (VAE) and information-theoretic objectives to learn time-aware causal representations. Experiments on synthetic and real-world datasets demonstrate the superiority of the proposed method over existing EDG techniques.\n\n## update after rebuttal\nI believe most of my concerns were solved by the authors during the rebuttal phase. So I tend to keep my score", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. Further analysis of spurious correlation caused by time factors and more concrete examples.\n\nR1. Thanks. Herein, we provide a formal characterization of the spurious correlation problem. In the proposed time-aware SCM, the time factor constitutes a latent confounder whose components $G$ and $L$ establish backdoor paths between static and dynamic spurious factors and the label, namely $Z_s^{st} \\leftarrow G \\rightarrow Y$ and $Z_s^{dy} \\leftarrow L \\rightarrow Z^d \\rightarrow Y$, thereby introducing spurious correlations. Under this configuration, naively maximizing mutual information between features and labels (that is, minimize the cross-entropy loss) will cause the model to learn spurious features. Specifically, the model may erroneously utilize brightness features with strong temporal correlations as discriminative cues for vehicle detection in Figure. 1, rather than learning essential shape semantics. To make the example more concrete, we provide more visualization results and analysis. Please refer to R2.\n\n> Q2. Improvement on visualization to tell the trained model learned the spurious correlations.\n\nR2. Thanks. To more clearly demonstrate the effectiveness of our method, we have enhanced the visualization and presented it in Fig. 2 of https://anonymous.4open.science/r/2150-1CBD. It can be found that in nighttime images, LSSAE [1] identifies the image as “No Car” based on lighting, while our method correctly focuses on the semantic information of the car, identifying it as “Car”. In daytime images, although LSSAE successfully classifies the image as car, it relies on environmental factors, whereas our method correctly identifies it based on the car's semantic information. Therefore, the trained model may learn the spurious correlations. \n\n[1] LSSAE, ICML 2022.\n\n> Q3. Comparison with methods using time factors to avoid spurious correlations.\n\nR3. Thanks. Time-series causal modeling constitutes the most pertinent research direction to this work. Most existing time series causal methods [2-4] construct temporal SCMs that incorporate causal factors and aim to learn causal representations based on the properties of these factors. However, due to the complexity of dynamic scenes, the behavior and nature of causal factors also become intricate, often requiring strong assumptions, such as the reversibility of the generation function and an additive noise model. \n\nIn contrast to them, meticulous deliberation has been conducted regarding the majority of factors in our method, wherein causal variables are explicitly decomposed into static and dynamic components for joint modeling. This modeling approach allows our method to learn complete causal representations by first learning easily obtainable static causal factors by and using them as anchors to learn dynamic causal factors, without requiring stringent assumptions, thus endowing it with extensibility to more complex scenarios. Furthermore, the integration of causal mechanism drift enables better adaptation to the underlying data distribution.\n\n[2] Causal-HMM, CVPR’21.\n\n[3] TDRL, ICLR’22.\n\n[4] CtrlNS, NeurIPS’24.\n\n> Q4. Visualization improvements and the idea for mitigating significant deviations from the trend.\n\nR4. Thanks. In the displayed visualization results, the similarity in the hot map positions may be attributed to the fact that the vehicle's position is directly in front of the camera. We have improved the visualization, as shown in Fig. 2 of https://anonymous.4open.science/r/2150-1CBD.\n\nWhen faced with unseen distributions significantly deviating from the training trends, the performance of EDG methods inevitably deteriorates, as they are designed under the assumption of slow and regular distribution changes. However, our approach incorporates the learning of static causal factors, enabling the model to maintain relatively stable generalization capabilities. To illustrate this, we randomly reorder the test domains of RMNIST and evaluate different methods. Specifically, we keep the training and validation sets unchanged, while rearranging the domains in the original test set, which previously arrived sequentially from 130°, 140°, ..., 180°, into 170°, 140°, 180°, 160°, 130°, 150°. The results in the table show that our method still outperforms the baselines. In addition, techniques such as out-of-distribution detection can be used to identify moments of significant distribution shifts and adjust the model to mitigate the impact of abrupt and violent shifts. \n\nFinally, we also conduct an experiment on the more challenging FMoW dataset as shown in R2 of Reviewer GuDA. it can be found that our approach remains effective in generalizing in more challenging scenarios, exhibits the potential for deployment in complex real-world applications.\n\n|| LSSAE | SDE-EDG | SYNC |\n| --- | --- | --- | --- |\n| RMNIST_Reorder (Wst./Avg.) | 35.7/40.8 | 28.1/42.0 | 38.1/43.4 |", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes,  I would say the datasets and the tasks are set correctly.", "theoretical_claims": "Yes, No clear mistakes are found in the proofs.", "experimental_designs_or_analyses": "Kinds of, I would say the datasets and the tasks are set correctly. and the results of each experiment also show the effectiveness of the proposed method. However, some visualization can still be improved.", "relation_to_broader_scientific_literature": "I think from what I know, and also the way the authors claimed, I think this method has a potential to be adapted for different/more extensive applications.", "essential_references_not_discussed": "I seem the authors attempted to discuss more on details, but there is still a lack of discussion, more details see questions listed.", "other_strengths_and_weaknesses": "Pros:\n1.\tThe experiments are sufficient, which visible performance improvements.\n2.\tThe method is technically sound, especially for considering the time factor of modelling causal reasoning. I think the idea is reasonable.\n3.\tThe functions and theoretical proving seems good.  \n\nCons:\n1.\tThe idea of using time factor for causal reasoning is reasonable, but not being expressed very clear. The background of why time matters for building correlations is still not clear enough. Directly saying that the time factor may cause spurious correlations, is not so convincible. but I think there is still space to even talk deeper, with more concrete examples. \n2.\tThe figure 1 can be improved with some day time images which can be recognized correctly, to show the difference between results of daytime and nighttime images. Then tell the trained model learned the spurious correlations.\n3.\tThe idea of capturing the time factor during causal reasoning is not the first time, what is the main difference between the proposal modelling and the existing methods who claim they also using time factors to avoid spurious correlations?\n4.\tFrom the shown visualization results, there is potential overfitting to temporal trends, e.g., every image shows the prediction hot map just in front of the camera.  The model might learn domain-specific time factors rather than true causal mechanisms, since we can not really tell from an un-explainable DNN. This may lead to reduced robustness when faced with unseen distributions that deviate significantly from training trends. Do you have an idea of how to conquer this?", "other_comments_or_suggestions": "Please check the questions above.", "questions_for_authors": "Please check the questions above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aAGsC9IrDl", "reviewer": "Reviewer_c4gP", "summary": "To solve the problem of poor evolving domain generalization caused by spurious correlation between data and targets across domains, this paper proposes a time-aware structural causal model with static-dynamic causal representation learning (SYNC). SYNC introduces mutual information to constrain the model to learn static-dynamic causal representations, produces good causal predictors by preserving intra-class compactness of causal factors both across and within domains. The results show that SYNC has better time generalization.\n\nThe author's response addressed most of my concerns about this article, so I update the score to weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. Modeling the drift factors may be not necessary.\n\nR1. We argue that modeling $Z^d$ is indispensable. Although both $Z^d$ and $Z^{dy}$ are influenced by $L$ and contribute to $Y$, $Z^{dy}$ operates in the feature space while $Z^d$ pertains to the label space which is designed to capture mechanism drifts. To be clear, $Z^d$ essentially characterizes the evolution of the classifier itself, which is crucial in EDG, as also emphasized in [1-2]. Furthermore, we conduct an ablation study on $Z^d$ and show result below, which clearly demonstrates the necessity of modeling drift factors.\n\n[1] LSSAE, ICML’22.\n\n[2] DDA, AAAI’23.\n\n||RMNIST|Portraits|Overall\n|-|-|-|-\n|SYNC|50.8|90.8|70.8\n|SYNC w/o $Z^d$|47.6|89.9|68.8\n\n> Q2. Regarding the posterior distribution model for $z_t^{dy}$.\n\nR2. We clarify that the posterior $q_{\\theta}(z_t^{dy}|z_{<t}^{dy},x_t)$ in our model is valid. \n\nIn VAE frameworks, the variational posterior $q_\\theta(z_t^{dy}|z_{<t}^{dy},x_t)$ is naturally learned through the inference network. The conditional dependency on $z_{<t}^{dy}$ reflects a data-driven modeling of temporal dynamics, independent of prior knowledge about $p(z_{<t}^{dy})$. Crucially, while both modeling $z_{<t}^{dy}$ and $x_{<t}^{dy}$ are reasonable, $z_{<t}^{dy}$ represents a ​low-dimensional encoding of $x_{<t}^{dy}$, compressed via LSTM hidden states to bypass high-dimensional data handling. This approach aligns with established sequence modeling practices [3], where latent variables effectively capture temporal dependencies.\n\n[3] S3VAE, CVPR’20.\n\n> Q3. About conflicting optimization goals.\n\nR3. We believe that the reviewer may have some misunderstandings. First, the misunderstanding fundamentally arises from a misinterpretation of the collider structure $Z^{st} \\rightarrow Y \\leftarrow Z^{dy}$, where $Z^{st} \\perp Z^{dy}$, $Z^{st} \\not \\perp Z^{dy}|Y$ hold. This is a well-established principle in causal science, commonly referred to as the collider effect. Therefore, these two objectives are reasonable. In addition, the second objective does not directly take the static and dynamic factors as input. Instead, it operates on the representations processed by the causalizer.\n\n> Q4. Misunderstanding about information leakage during inference.\n\nR4. We respectfully disagree with the reviewer's view. During inference, static and dynamic causal factors are obtained by corresponding VAE encoder and causalizer, and the drift factors are inferred solely from the learned prior $p(z_t^d|z_{<t}^d)$, without access to ground-truth labels. Hence, the concern about possible information leakage may be misplaced. Please refer to Appendix. D for full details about training and inference procedures of our approach.\n\n> Q5. Regarding for showing results in domains.\n\nR5. We stress that EDG is inherently designed for future-domain generalization. Accordingly, following the common practice adopted by most existing EDG methods such as [1, 4], we evaluate on the last third of the domains to assess this capability. However, to address potential concerns, we also report the average test performance across all domains on Circle and RMNIST, demonstrating the superiority of SYNC.\n\n||MMD-LSAE|SDE-EDG|SYNC\n|-|-|-|-\n|Circle|92.3|91.1|93.0\n|RMNIST|81.2|81.7|82.0\n\n[4] SDE-EDG, ICLR’24.\n\n> Q6. Lack of sufficient comparison with other baselines in Figures 5 and 6.\n\nR6. For Fig. 6(a-b), LSSAE [1] is chosen as the baseline since other methods don't explicitly model both static and dynamic factors. Although MMD-LSAE, an extension of LSSAE, considers these factors, it models them deterministically, making mutual information (MI) computation infeasible under our MWS-based estimation scheme. For the remaining figures, we have added the optimal baselines and redrawn them, as shown in Fig. 3 and Fig. 4 of https://anonymous.4open.science/r/2150-1CBD. The results demonstrate that SYNC still outperforms other methods, as claimed.\n\n[5] MMD-LSAE, TPAMI'23.\n\n> Q7. Clarification of this work's contribution to resolving spurious correlations.\n\nR7. Our method disentangles static and dynamic causal factors to learn time-aware causal representations and address spurious correlations. It works as follows: First, MI $I(z_t^{st},z_t^{dy})$ is minimized to disentangle the factors. Then, Proposition 3.3, Lemma 3.4, and Lemma 3.5 guide the learning process. Specifically, static causal factors can be identified by maximizing conditional MI between static components of consecutive domains given the class (Eq. (4)). Dynamic causal factors can be obtained by anchoring on static causal factors and maximizing conditional MI within the same domain given both the class and static causal factors (Eq. (8)). Based on these theoretical results, we use causalizer modules to extract finer-grained static and dynamic causal factors by optimizing the above objectives.\n\n> Q8. Regarding for some symbol errors.\n\nR8. Thanks. We will fix these errors and thoroughly revise the manuscript.", "claims_and_evidence": "Most of the claims in this paper are clear and convincing, but I still have some concerns:\nModeling the drift factors $Z^d$ may not be necessary. In the structural cause model, the drift factor $Z^d$ and the dynamic factor $Z^{dy}$ are very similar. They are both affected by the local variable L and they both affect the label Y, which means that their distributions both shift as L changes, and $Z^d$ can be viewed as a dynamic factor.", "methods_and_evaluation_criteria": "1. In Eq. (1), $z_{<t}^{dy}$ is the condition for the prior distribution and the posterior distribution of the dynamic factor $z_t^{dy}$, but we do not know the prior distribution $p(z_{<t}^{dy})$ when we infer the posterior distribution of $z_t^{dy}$. I think the true posterior distribution of $z_t^{dy}$ is $q_{\\theta}(z_{t}^{dy}|x_{<t})$ or $q_{\\theta}(z_{t}^{dy}|x_{≤t})$.\n2. Eq. (3) minimizes the mutual information between static and dynamic representations to disentangle $z^{st}_t$ and $z^{dy}_t$, but Eq. (8) maximizes their mutual information to align static and dynamic representations of the same category within the domain. These two goals seem to be in complete conflict. \n3. Eq. (11) uses the label $y_t$ to learn the drift factor $z^d$, and then $z^d$ is used to predict the label $\\hat{y}_t$. There can be serious information leakage, because the true label and the predicted label may be directly correlated by the model. In extreme cases, the drift factor learned by the model may be invalid.", "theoretical_claims": "I have checked the proofs of Proposition 3.3 and Theorem 3.6 provided in this paper, and the proof of Theorem 3.6 May be wrong. Taking the true label as the model input can cause the true label and the predicted label to be directly associated by the model and cause the model to learn invalid representations.", "experimental_designs_or_analyses": "I have checked the experimental designs and analyses of this paper, including the predicted performance comparison experiment of all algorithms on all datasets, and the ablation experiments of SYNC on the RMNIST dataset. This paper also shows the decision boundary of the algorithms, the change of decision accuracy in different domains and the independence of dynamic and static representations. Although the authors present a large number of results, I still have some concerns:\n1. This paper only shows the results of the algorithms in the last few domains. The results of the algorithms in all domains need to be compared to demonstrate the generalization ability of SYNC in different domains.\n2. Figures 5 and 6 compare the results of only a few algorithms, and these algorithms are not the best baselines for the corresponding datasets. Please show the results of other baselines, or explain the reasons for presenting results from only those baselines.", "relation_to_broader_scientific_literature": "This paper attempts to solve the problem of modeling spurious associations between data and targets across domains. Prior to this, LSSAE and MMD-LSAE had attempted to model evolving patterns.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths：\n1. This paper is important for the study of evolving domain generalization. It introduces causal learning to solve the problem of spurious correlation between data and targets modeled by existing models. \n2. This paper is innovative to some extent, introducing mutual information to constrain the model to learn dynamic and static representations.\n\nWeakness：\n1. The contribution of this paper to solving spurious associations is limited. Although this paper proposes a structural causal model that divides potential factors into causal and spurious factors, this paper uses mutual information to constrain the model to disentangle dynamic and static representations, without disentangling causal and spurious factors. Therefore, whether the proposed model solves the spurious association needs further proof.\n2. The method in this paper is not reasonable. For example, the posterior distribution of $z^{dy}_t$ in Eq. (1) is conditional on unknown variables, Eq. (3) and (8) conflict, and Eq. (11) has label information leakage. \n3. The experimental results in this paper are incomplete. Authors should present the results of algorithms in all domains of the dataset, and should highlight the results compared to the optimal baseline under different datasets.", "other_comments_or_suggestions": "1. There are some symbol errors in the paper that need to be corrected. For example, $Z^{gc}$ and $Z^{lc}$ in the formula below definition 3.1 and “variant E” in the ablation study.", "questions_for_authors": "1. Can the author explain in more detail how DYNC disentangles causal factors and spurious factors? I think the author overemphasizes disentangled dynamic and static representations and ignores the more important goal, which is to eliminate the spurious associations between data and targets.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces one method for improving temporal generalization in evolving domains by explicitly disentangling static and dynamic causal representations. The authors have provided comprehensive responses to the reviewers' concerns through effective revisions that significantly enhance the paper's clarity and argumentative depth. After revision, the paper provided sufficient experiments and good visible performance. Overall, considering the core strengths and drawbacks of the proposed method, I recommend accepting this paper for publication.
Decision: Accept (poster)