Evolving Standardization for Continual Domain Generalization over Temporal Drift
Authors: Mixue Xie
Shuang Li
Longhui Yuan
Chi Harold Liu
Zehui Dai
Venue: NeurIPS 2023 poster (2023)
Keywords: domain generalization
sequential learning
temporal drift
feature standardization
TL;DR: We formulate the problem of continual domain generalization over temporal drift and propose an evolving standardization method for this problem.

Abstract: The capability of generalizing to out-of-distribution data is crucial for the deployment of machine learning models in the real world. Existing domain generalization (DG) mainly embarks on offline and discrete scenarios, where multiple source domains are simultaneously accessible and the distribution shift among domains is abrupt and violent. Nevertheless, such setting may not be universally applicable to all real-world applications, as there are cases where the data distribution gradually changes over time due to various factors, e.g., the process of aging. Additionally, as the domain constantly evolves, new domains will continually emerge. Re-training and updating models with both new and previous domains using existing DG methods can be resource-intensive and inefficient. Therefore, in this paper, we present a problem formulation for Continual Domain Generalization over Temporal Drift (CDGTD). CDGTD addresses the challenge of gradually shifting data distributions over time, where domains arrive sequentially and models can only access the data of the current domain. The goal is to generalize to unseen domains that are not too far into the future. To this end, we propose an Evolving Standardization (EvoS) method, which characterizes the evolving pattern of feature distribution and mitigates the distribution shift by standardizing features with generated statistics of corresponding domain. Specifically, inspired by the powerful ability of transformers to model sequence relations, we design a multi-scale attention module (MSAM) to learn the evolving pattern under sliding time windows of different lengths. MSAM can generate statistics of current domain based on the statistics of previous domains and the learned evolving pattern. Experiments on multiple real-world datasets including images and texts validate the efficacy of our EvoS.

Evolving Standardization for Continual Domain Generalization over Temporal Drift
Mixue Xie
Shuang Li Longhui Yuan' Chi Harold Liu'
Zehui Dai²
Beijing Institute of Technology, China LLazdda Search & Monetisation Tech, China
(mxxie, shuangli, 1ow.gooynnnnn @bit. edu. en, liuchi020gmail.. com
zehui
com
Abstract
The capability of generalizing to out-of-distributin data is crucial for the deploy- ment machine learning models in the real world. Existing domain generalization (DG) mainly embarks on offline and discrete scenarios, where multiple source domains simultaneously accessible and the distribution shift among domains is abrupt and violent. Nevertheless, such setting may not be universally applicable to all real-world applications. as there are cases where the data distribution gradually changes over time due to various factors, e.g., the process of aging. Additionally, as the domain constantly evolves, new domains will continually emerge. Re-training and updating models with both new and previous domains using existing DG meth- ods can be resource-intensive and inefficient. Therefore, in this paper, we present a problem formulation for Continual Domain Generalization over Temporal Drift (CDGTD). CDGTD addresses the challenge of gradually shifting data distributions over time, where domains arrive sequentially and models can only access the data of the current domain. The goal is to generalize to unseen domains that are not too far into the future. To this end, propose an Evolving Standardization (EvoS) method, which characterizes the evolving pattern feature distribution and mitigates the distribution shift by standardizing features with generated statistics corresponding domain. Specifically, inspired by the powerful ability of trans- formers model sequence relations, we design multi-scale attention module (MSAM) to learn the evolving pattern under sliding time windows of different lengths. MSAM can generate statistics of current domain based on the statisties of previous domains and the learned evolving pattern. Experiments on multiple real-world datasets including images and texts validate the efficacy of EvoS.
Introduction
In real-world applications, the assumption that training and testing data conform to the same distribu- tion, a prerequisite for the success of contemporary deep learning methods, is seldom valid. A prime example of this can be found in autonomous driving, where a vehicle may traverse environments that switch from daylight to nightfall or from urban to rural. As environmental conditions change, the issue of distribution shift [5 4 51 62) arises. Moreover, directly utilizing model trained (OOD) context often results in catastrophic perfor-
in-distribution (ID) data in an mance deterioration. Consequently, ensuring that models perform well on OOD data has emerged as a crucial challenge for the widespread deployment machine learning models in the real world.
To cope with the distribution shift, two dominant paradigms have been systematically explored depending on the availability of target (test) domain. One is domain adaptation (DA), which aims to assist the model learning on unlabeled or label-scarce target domain by transferring the knowledge
"Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Yet, target data are not always known or available in advance. The other is domain generalization (DG), the goal of which is to learn a model capable of generalizing to any unseen domain by using date from multiple related but distinct source domains 66 27 63 26] Though the second paradigm DG has attained some encouraging outcomes, its configuration is limited to offiin and discrete scenarios where multiple source domains can be accessed simultaneously, and the distribution shift among domains is sudden and severe. For instance, the prevalent benchmark PACS [25] in current DG methods comprises four distinct domain styles- - "Art", "Cartoon", "Photo" and "Sketch". Nevertheless, this type of configuration may not be suitable for all real- realdworld applications. There are also some cases that the data distribution gradually evolves over time, the distribution shift arising from which is referred to as temporal drift [56].
from a related and label-rich source domain [ 34 29 (3].
On one hand, the real-world scenarios often exhibit underlying evolutionary patterns (59 44 For example, annual or monthly weather data can be utilized for weather forecasting [12 The evolutionary patterns can be exploited to enhance generalization capabilities towards future unseen domains that are not too distant. However, current DG methods often fail to consider these patterns, leading to suboptimal performance. On the other hand, since data distribution is constantly evolving, new domains will continue to emerge. Consequently, it is imperative to efficiently utilize these new domains to further enhance the model's performance for practical applications. For instance, in the context of advertisement recommendation, user browsing data for various products continually surfaces. How can we leverage these newly collected data to enable more accurate advertisement recommendations tailored to each user's preferences in the days to come? One simple approach is to store data from previous domains and retrain the model using both the new and previous domains with the existing DG techniques. However, such way may consume huge training resources due to the accumulation of data and is of low efficiency, especially for scenarios with rapid data accumulation.
In this paper, we formulate the aforementioned problems as Continual Domain Generalization over Temporal Drifts (CDGTD), where the data distribution gradually drifts with the passage of time and the domain arrives sequentially. And the goal of CDGTD to generalize well on future unseen domains that are not too far under the circumstance that only current domain is available at current times point while data from previous domains are inaccessible. Although some temporal / evolving DG methods [44 59 3 41 have been proposed to handle the temporal drift, most of them [59 44
works in non- intremmettt setting, i.c., multiple domains can be accessed simultancously. Instead, we design an Evolving Standardization (EvoS) approach specialized for CDGTD.
For CDGTD, there are two main challenges: how to characterize the evolving pattern the incremen- tal setting and how to achieve generalization using the learned evolving pattern. For the former, we draw inspiration from the sequence modeling capability of transformers [31 and design a multi-scale attention module (MSAM) to learn the evolving pattern underlying the feature distribution of domains. Specifically, we store the statistics (i.e., mean and variance of features) of previous domains and use sliding time windows of different lengths over the statistic tokens to obtain multi-scale information. Then multi-scale statistic tokens are fed into MSAM to generate statistics of current domain. MSAM is trained over the whole sequence of domains to learn the evolving pattern. Here, we integrate multi- scale information, with the consideration that some evolving patterns may be better characterized at different time intervals, e.g., seasonal climate and daily weather. For the latter challenge, in order to mitigate the temporal drift, each domain is transformed into a common normal distribution by conducting feature standardization with the generated statisties of corresponding domain. considering that the feature encoder may suffer from catastrophic forgetting and overfitting to current domain, we constrain it to learn a shared feature space among domains via the adversarial learning.
Besides,
1) We formulate a promising but challenging problem of continual domain general- ization over temporal drift (CDGTD), which has seldom been explored, compared with traditional DG. 2) An evolving standardization (EvoS) approach is specially proposed for CDGTD, which can characterize the evolving pattern and further achieve generalization by conducting the feature standardization. 3) Experiments on multiple real-world datasets with different models verify the effectiveness of EvoS and the flexibility to be applied on different models.
Contributions:
2 Related Work
Domain Generalization (DG) aims to learn a model that can generalize well to any unseen target domains by leveraging data from multiple source domains. In recent years, wide range of DG
2

---

methods have been proposed [6 39 27 26 63 19 37] According to the strategies of improving gen- eralization, existing DG methods roughly fall into the three categories. Representation learning based methods 7
2 17 42 aim to learn domain-invarian or domain-shared representations to enable models to generalize to unseen target domains. Data augmennttioon/generaiin based meth- 32 focus on manipulating inputs to facilitate learning general representations.
39
50
27
27
ods 55 61 Differently, other DG methods instead employ general learning strategies like meta-learning [26] and ensemble learning 221] to improve generalization ability. With full access to all source domains, these DG methods have achieved promising performance on unseen domains with abrupt and violent distribution shifts. Unfortunately, without the consideration of the evolutionary pattern of domains over time, existing DG methods are usually less efficient under the scenario of temporal drift.
63
57
Continual Learning (CL) is machine learning paradigm, where models learn continuously from a stream of tasks over time and meanwhile try to retain performances on all seen tasks 22
24 99 46 47 Existing CL methods can be roughly categorized into replay-based |46 10 52
regularization-base of CL and DA has attracted lots of attention [23
[22 60 30 and structure-bases
45
14 methods. addition, the combination
Yet, combining CL and DG remains underdevcloped. In this work, we propose CDGTD which continually trains the model on sequential domains, but the goal is to generalize well on novel domains in the near future. This distinct objective from CL yields different challenges: the modeling of underlying evolutionary patterns of temporal domains and how to utilize these patterns to mitigate the distribution shifts in forthcoming domains.
33
43 55).
Test Time Adaptation (TTA) focuses on adapting the source-pretrand purtreepetr model during testing phase with test data [40] 49 20
54
58
For example, 4 corrects the activation statistics of batch (0 uses a pack-propagation--t
manner to adjust only the weights of the linear classifier during test time. By contrast, CDGTD optimizes the model during training phase with sequential domains and emphasizes on generalization without using test data.
normalization using test data. T3A
Evolving/ / Temporal Domain Generalization has emerged in recent years, aiming to tackle the problem of generalizing on temporally drifted domains, where the environment evolves dynamically over time [41 B 44 159. GI [4I] proposes time-sensitive model architecture to capture the time-varying data distribution and the model is supervised with the first-order Taylor expansion of the learned function to advance the generalization in the near feature. DRAIN 3 launches a Bayesian framework to model the concept drift and utilizes recurrent neural network to dynamically generate network parameters to adapt the evolving pattern of domains. LSSAE 44 incorporates variational
inference to explore the evolving patterns of covariate shift and label shift in the latent space. The goal of learning evolutionary patterns from source domains and generalizing to domains in the near future is similar to our EvoS. The main difference is that the model in our EvoS is incrementally trained on sequentially arriving domains, considering the low efficiency of "offline" training with accumulated domains. By contrast, the aforementioned methods 41 require multiple source domains to be simultancously available. Besides, the Taylor expansion in [41), variational inference in (44) and network parameters generation in (3) make them hard to expand on large models.
3 Evolving Standardization for Continual Domain Generalization over Temporal Drift
3.1 Problem Formulation of CDGTD
Here, we take the C'claas classification problem as an example, where X and Y denote the data and label space, respectively. Suppose that T source (training) domains {D=, D2 .. ,..,." sequentially arrive, which are sampled from distributions at T different times points ( < 12 < ...... < tr. At time point f, L E (1, 2, ... T), only the domain D {ar yf N accessible, where nt E X, y E Y and N is the number of training samples in Dt. Previous and future domains are unavailable. In addition, as previous temporal/evolving DG methods [44] 41 3) we further assume that the data distribution P(X, Y) of domains evolves temporally, i.c., the distribution of domains changes along the time following certain patterns. The goal of CDGTD is to enable the model, composed of a feature encoder & : x - f E Rd (dy is the dimension of features) and classifier C ff y E 1....... C- 1}. to generalize well on K unseen target (test) domains in the near future: [DD-*JK 1- To this end, two main challenges need to be addressed. One is to characterize the evolving pattern of domains, which we address by designing multi-scale attention module (MSAM)
3

---

ateialic
memory
nacocosible gradient Zi, 5000 grdient cinn
G T.
somolo from lasture eirbuttons
average operation: Sigee
posl El.7
D
attention .
attention Ah
attention module
mutsi head self attention module A...
(MSAM)
D
average operation:
Yaaim 3%
disciimrator
D
stallsic
feature encoder
Feature Standardization
cloesifer
concatonated
F
espLa
!slding Sime window
token attention A. Figure 1: Overview of EvoS. Memory pool M stores previously generated domain statistics (i.e., J (mean) and a' (logarithm of standard deviation) features, S St 1). With the t-th domain D' available, loss Stan is minimized to train the multi-scale attention module (MSAM) generate statistics fi' and a' that approach the real ones via leveraging historical statistics in M. Besides, we sample features from historical feature distributions as the proxy of previous domains to conduct adversarial training with current domain, encouraging to learn a shared feature space.
inspired from the sequence modeling ability of transformers. The other is the generalization, which is realized by transforming domains into a common normal distribution via the feature standardization.
3.2 EvoS: Evolving Standardization
Single-scale Attention. Inspired by the capability of the transformer to model the relationships among sequences (3T), we introduce attention mechanism to model the evolving pattern among sequential domains. Before presenting our multi-scale attention module (MSAM), we first introduce how single-scale attention works. Without loss much generality, we assume that the feature distribution of domain 2 follows the normal distribution, which is characterized by the mean vector μ¹ E Rdd and the standard deviation vector o¹ E Rdd, Now, our goal turns out to be learning the evolving pattern of the feature statistics (i.e., μr¹ and o'). In practice, ensure that of is non-negative, we choose to learn a' (the logarithm of o'). In this way, o¹ = exp(a') (a) is always non-negative. store the learned statisties time
Besides, considering the unavailability of previous domains, point & into a memory pool M for future uses once the training procedure finishes on domain domains ready, then utilize the multi-head
D'. Having the feature statistics of previous self-attention module A to generate the statistics of domain D' at time point 4.
Coneretely, given statistic tokens { of previous the token of concatenated statistics from domain Dí, the output of A expressed as
domains, where s¹ [Mu', at) E R2d, is
S = A(5 = [SA, (AA ".., SAm, (
W Wfe R
= 8
st s
St-1 12 (t-1)x
where & is the i-th output statistic token of A, n4 and d, are the number and feature dimension of heads in the multi-head self-attention module and W fo is the learnable parameters of A to convert feature dimensions. SA,(.) is the self-attention of the i-th head, which operates as follows:
[ st-1 st 5 W
gku: w
Wi aku R 2d, x3d
(At = softmax(S - /
(2)

---

where W oku is the learnable parameters of the i-th head and Sig S S,²"" are the query, key and value embeddings of 5 Finally, we use the average of all output statistics tokens of the attention module A as the generated statistic token 8² for time points t;
[a [..aa]]= = = aoog(
1 == - 8"
(3)
By continuously generating new statistics from historical ones throughout the whole sequence of domains, the attention module A is expected learn the evolving pattern of feature distributions. Note that we directly introduce learnable statistic vectors ğ a' and µi a² for time points and (g, respectively, since the tokens of historical statistics are not enough for the attention module A to work. That is, the attention module A is only used at time points 4, 2 3.
Multi-scale Attention Module (MSAM). The above details how model the evolving pattern using single-scale attention. However, it is limited to learn the pattern using sliding time window of length I over the statistic tokens. Sometimes, the evolutionary pattern may be better captured by considering longer time intervals. For instance, the data of season in each time window would be more suitable for capturing the evolving pattern of seasonal climate, rather than data of a single day in each time window. Considering this, we introduce multi-scale attention in Fig. which leverages information from observation windows of different lengths to better model the evolving pattern.
Specifically, for the multi-head self attentonn module A. responsible for the time window of length w, we slide the time window with stride over historical t statisties tokens and concatenate the statistic tokens in each window to obtain the input 3 for the attention module An at time point &,
8 is
Sy E ERw-2d, (Isist- (t w 2
5t = 13 wi 10: 10
S St1 E (((-))((wxd)))
(4)
And similarly, we use the average output x(-w+1 of the attention module An as the generated statistics for the sliding time window at the next time point, the formulation of which is expressed as
Stt = (5' = [18 bai
5 E (w)
w
w+1 ---w+1 g)
= aug( (5th)) t- w
wt- E Rw-24
(5)
Note that in MSAM, the attention module Aw is to predict the statistics in the next sliding time window of length w. instead of one statistic token as in the single-scale attention. This may encourage the attention module to also capture the domain relationships within the window.
Then, we split g--++1 into w parts: [st. =+1 st-1, s.y.1
g where s/ RRdd can be regarded as the predicted statistic token for time point t, using the attention module Aw. j = t t- ur + 1, .... L 1, f. Finally, the generated statistic token & for time point & using MSAM is denoted the average of predicted statistic tokens for time point & at different time window lengths:
Σ
8
where W is the maximum length of the sliding time window. Generally, MSAM integrates evolving patterns learned at different scales, contributing better estimation of future feature distributions.
Feature Standardization. For CDGTD, the second main challenge is generalization. To this end, we leverage the generated feature statistics by MSAM transform the distribution of corresponding domain into a standard normal distribution, by which the temporal drift is mitigated. Concretely, for feature f E(x)), its standardized feature 2 via the feature standardization formulated
exp(a')
Another benefit of feature standardization that it allows classifier to be trained on common feature distribution, thus avoiding the problem of overfitting to current domain and catastrophic forgetting.
5

---

3.3 Model Training
This section outlines the four losses Ci C'ston from and Cady involved in the training stage at time point 4. The first one is the essential loss to supervise the model learning for specific tasks, e.g., the following cross-entropy loss Ce for classification tasks in this paper.
Σ CEEpl. p = softmax ( (c(E)T99(7)))
Nº
min C.e = E.C
(8)
N
where .) is the cross-entropy. Here, stopping gradient sg(-) .) is adopted to stabilize training.
Losses L' La and Leom are responsible for the training of the attention module A. at time point f W = 1,,,..., W. The former loss Castn in Eq. 10 minimized to ensure that the standardized feature z follows a standard normal distribution. Specifically, we first calculate the mean vector and variance vector of the standardized features as
1 Nr N
sg(f)) - P. exp(d")
N2
sg(f)) p exp(a')
mean f
variance : N i=
O'
4²) (9)
i=1
Then the loss St at time point & is expressed
min Citan = |1 0|12 + 11 1| â"
1st52
(10)
min C.tan = l1 0|12 + |1 1|12. 2 +
A...
where y is the dimension of features and the length of the sliding time window. Here, we just want the learnable/generatedd fi' à' to approach the statisties of true feature distribution, so stopping gradient operation sg(-) is conducted on f to prevent the gradient of L flowing to the feature encoder E. Otherwise, the gradient of Ca may distort the feature distribution. Ideally, if Cttan is minimized, the generated statistics ç and exp(â") by our MSAM would be equal to the mean vector and standard deviation vector of the true feature distribution of D This means that our MSAM can predict future feature distributions based on historical feature statistics.
And the loss C.om is to ensure the consistency between generated statistic tokens in the next sliding time window and real statistic tokens. Specifically, we calculate L for attention module A as
min A.
L
| - w+t s g 11,,
A 2,
(11)
w
where 8/1 and point ly using the attention module Aw and the real statistic token from memory pool M. This loss is expected to also capture the evolutionary pattern within the time window.
+ ,1,...,,-....... 1 are respectively the predicted statistic token for time
-6-
The fourth loss L' is introduced with this consideration that the feature extractor & may overfit to current domain and lacks generalizability. To avoid this, we additionally conduct adversarial training between the feature encoder & and discriminator D. Nevertheless, historical data unavailable. So we instead resort to the learned/generated statistics at previous time points. Concretely, we randomly sample batch of B samples {f"T: ............. f from each distribution M(H"". (exp(a"")) m 1,2,... 1, via Eq. [2 each iteration, and use them as the proxy of historical domains.
S a, where a is a truncation hyper-parameter to control the sampling area. Then discriminator D is trained to distinguish [ from {f ..... f 1 and the feature extractor tries to confuse D. Such adversarial process is achieved by the following loss Ca
f 10" Th exp(a""). e e~ A
(12)
N
max min Ci 3
-
log(D
Σ
log(1 - D(F)))) t 2. (13)
Bx
NO
In practice, the gradient reverse layer (GRL) 15 is used achieve the adversarial training. To sum up, at time point &,, the model is trained to minimize the following total loss Clotaii
W
Ciotel C. + Ci + Etho
Ltu + X
(14)
wmml

---

Table 1: Accuracy (%) on Yearbook, RMNIST and fMoW. CDGTD setup are bolded and underlined. (Yearbook: K 5, RMNIST: K 3, fMoW: K 3)
The best and second-best results in
Ycarbook
RMNIST
IMoW
Incremental training
Access multiple domains
Accuracy (96)
Accuracy (%6 T
Accuracy (%%
Method
Conference
2>7++
DOD
OOD
DD
OOD
OOD
2>++1
0OD
OOD
avg.
worst
avg.
worst
avg.
worst
Offline
89.30
88.46
86,81
98.15
92.14
83.89
72.45
59.76
49.85
19
95.10
85.05
64.77
54,92
46.51
CORAL
30
ECCV Workshops 16
X
95.94
91.79
88.84
93.04
79.10
62.96
62,
51.42
42.19
Mixup
ICLR
94.98
91.12
88.35
97.11
89.66
79.63
70.27
57.73
48, 48.04
LISA
57
JCML'22
95.51
92.97
91.29
96.21
87.04
75.15
70.05
55.52
44.61
CDOT
43
arXiv'19
<
95.17
92.90
46
97.
90.19
79.67
CIDA
55
ICML'20
92.36
97.43
89.19
GI A1
NeurlPS'21
97.42
96.37
95.73
97.78
91.00
82.46
61.62
50.83
42.78
LSSAE 44
ICML'22
93.93
92.12
85.75
96.73
90.36
$2.13
59.15
48. 66
41.38
IncFinctune
X 96.61
94.72
93.48
98
92.80
S4.61
65.52
$3.99
45.23
Mixup [6]
ICLR/18 18
X 90.21
89.83
88.43
98
92, 28
83.45
64.84
52.00
42.54
SinCLR
ICML'20
95.94
98
90.98
44.71
SwAV
NeurlPS 20
X 97.37
94.27
44
98 08
90.85
80.96
66.47
54.51
45.29
EWC 22
arXiv'16
X 97.18
95.12
93.64
98.56
92.02
$2.80
66.23
54.55
45.80
SI
ICML'17
X 97.09
94.67
93:48
98.61
93.27
85.65
66.61
$4.89
46.46
A-GEM 10
ICLR'19
X
94,36
90.96
85.88
95.99
86.95
75.45
54.54
47.61
41.13
SGP
AA
88.97
78.05
DRAIN 3
ICLR'23
X 96.23
94.71
93.73
98 98.52
93.09
85.75
67.22
55.05
46.24
Evo8
X
97.37
95.53
94.78
98.64
93.84
87,04
67,18
54,64
45.86
For IM backbone DenseNet-121 121 to apply full GI and DRAIN. So apply DRAIN only to classiser and apply Gl without the fine luning stage.
where À is a is finished, we store the learned/generated statistics of current domain 2 into memory pool M by
to balance the loss tradcoff. Once the training procedure at time point
4 a' < a"
(15)
In the inference stage, we use MSAM to generate future statistics g based on the statistics {st) [** in memory pool M and store it into .M as Eq. 15 for the generation next time point.
Due to space limitation, the training and testing procedures are provided in the appendix.
4 Experiments
4.1 Experimental Setup
Thanks to the work in (56). several real-world datasets with distribution shifts over time have been available. And we evaluate EvoS on three image classification datasets (Yearbook and fMoW from (56) and RMNIST) and two text classification datasets (Huffpost and Arxiv from (56)). Yearbook collects data from 1930 to 2013, where we treat every four years as a domain and use the first 16 domains for training (T 16), the last 5 domains for testing (K 5). fMoW includes data of 16 years and we set T = 13, K 3. RMNIST contains 9 domains with T 6, K = 3. Huffpost includes data of 7 years with T 4, K 3. Arxiv collects data of 172 categories for 16 years, with T = 9, K = 7. For each training domain of all datasets, we randomly select 90% data as training split and 10% data as validation split. Following the backbones
(56), we use 4-layer convolutional network (56) for Yearbook, DenseNet- 121 [88 pretrained on ImageNet for fMooW, pretrained DistilBERT base model 48) for Huffpost and Arxiv, and the ConvNet in 44 for RMNIST. For each attention module A. in MSAM, its dimension of head d, is set to 8 and its number of heads set to w пd Specifically, n4, is set to 16 for Yearbook, 32 for RMNIST, 64 for fMoW and 128 for Huffpost and Arxiv. For optimization, we use the Adam optimizer with Ir 1e 3 for Yearbook and RMNIST, = 2e 4 for fMoW and = 2e 5 for Huffpost and Arxiv. The batch size is set to 64 for all datasets. As for hyper-parameters, we select them via grid search using the validation splits of training domains and finally use a = 2.0 for RMNIST, a = 1.0 for others, À 1.0, W = for all datasets. For all tasks, we report the mean of 3 random trials. Due to space limitations, please refer to appendix for more details. Code is available at hhtps:::/ :// //tg//gtttttttt com com/BIT
4.2 Main Results
In addition to the incremental training scenario, we also provide results in non-incremental scenario with all source domains simultaneously available, which serves as upper bounds. Among compared baselines, "Offline" denotes merging all source domains into one domain and training the model with the merged domain, while "IncFinetune" represents incrementally training the model in domain- by-domain fashion. For each dataset, report the accuracy the nearest target domain (DI+1),
7

---

Table 2: Accuracy (%) on Huffpost and Arxiv. The best and second- results in CDGTD setup are bolded and underlined. (Huffpost: =
3, Axriv: K 7)
Huttpost Accuracy (%)
Arti Accuracy (%)
Incremental training
Access
Method
Conference
multiple
domains
D
O01>
OOD
DJ
OOD
avg.
worst
avg.
worst
Othine
72.74
71.50
09.63
57.49
52.38
49.28
IRM 2
arXiv' 19
71.04
70.31
68.97
51.11
45.89
42.86
CORAL 50
ECCV Workshops 16
71.34
70.08
68.68
50.98
45.77
42.71
Mixup 6
ICLR'18 ICML'22
X
73.34
71.16
69.29
57.58
52.77
49.62
LISA
72.19
70.24
68.60
56.53
49.67
G1 411
NeurIPS'21 11
d
68.06
66.32
64.64
53.43
49.19
46.13
Incl ineluue
X
73.57
71.98
69.80
56.22
32.43
49.37
Mixup 61
ICLR'18
73.07
71.52
69.44
56.64
52.95
49.97
BWC 22
atXiv' 16
X
73.64
71.53
68.99
56.60
52.78
49.73
S1 600
ICML'17
72.58
71.50
69.61
49.98
47.27
44.77
A-GEM 10
ICLR'19
72.23
71.16
69.10
52.02
48.91
46.03
DRAIN 3
ICLR'23
X
73.42
71.75
69.69
56.04
52.07
48.97
73.42
72.36
70.19
$6.60
50.19
For Hullipost and Arxiv, backhone DistiBERT-bass buse is too big to apply the full and DRAIN. we apply DRAIN only to the classifier and apply Gll without the fine-tuning stage.
the average and worst accuracy of future K domains ("OOD avg.": K Σ Acc(DD*k), "OOD worst": (1,2k .K) *c)(
Results on image classification tasks are provided in Table 5 where we can observe that EvoS consistently achieves superior performance across three reported metrics on Yearbook and RMNIST datasets, compared with the methods incremental training scenario. Especially, Evoš outperforms DRAIN by over 1% according the metric "OOD worst" on Yearbook and RMNIST, which means that our EvoS learns more robust evolving patterns. addition, we notice that conventional DG methods IRM (2] and CORAL [50] perform worse than temporal DG method GI [4] Yearbook and RMNIST, showing the importance of leveraging evolutionary patterns for generalization over temporal drifts. Though being effective for the small networks used in Ycarbook and RMNIST, the advantage of GI disappears on fMoW dataset with backbone DenseNet-121, where its first- order Taylor expansion requires extremely huge computing resources (over 80G). And our available resources cannot afford it. When ablating the Taylor expansion, the performance GI on fMoW is not the best. So GI is hard expand on larger models and the similar issue also exists in DRAIN and LSSAE. Finally, we find that for dataset (MoWW not only our method performs unsatisfactorily but also other methods, except for the "offline" method (i.e., merging all source domain into one domain to train the model). We infer this may be due to the temporal drift is not well presented in this dataset.
Results on text classification tasks shown in Table 6 where EvoS consistently achieves the best performances according to the average and worst accuracy of futurre K domains under the CDGTD setting. This can be owed to the more robust evolving patterns captured by our multi-scale attention module (MSAM). Similarly, IRM and CORAL show obvious performance drop compared with DRAIN, once again demonstrating the necessity to learn evolving patterns for the problem of CDGTD. And the large performance gap between continual learning methods (i.e., SI and A-GEM) and our method EvoS on dataset Arxiv verifies that our method EvoS fully utilizes the historical knowledge to learn evolutionary patterns, while SI and A-GEM do not consider this.
4.3 Analytical Experiments
Ablation Study, Firstly, we study the influence of the stopping gradient sg(-) in the losses L and L'ttnn- The results of variant A and B in Table almost degenerate to random predictions,
compared with variant C. This suggests that using sg(-) in loss Ct St is essential. Otherwise, the gradient of Lt stan would largely distort the learning of feature en- coder, causing training collapse. Mean- while, the inferior performance of variant C to EvoS also indicates that the gradi- ent of C. should not interfere with the learning of MSAM. Secondly, we ablate the losses Cedr and Lte to testify their necessity. We can see that the results of
Table Ablation study of EvoS on dataset Yearbook.
Method
Scale
Loss
Using
sg(.)?
Using
00D
single multi
L.
Litan
truncation a?
avg.
Variant A
No
No
Yes
51.28
Variant
Yes
No
Yes
55.37
Variant
4
No
Yes
Yes
93.97
Variant D
Yes Yes
Yes
93.25
Variant E
4
Yes
Yes
Yes
94.75
Variant F
/
v
Yes
Yes
Yes
94,27
Variant G
Yes
Yes
No
94.46
EvoS
Yes
Yes
Yes
95.53
Variant H
Yes
Yes
94.09
8

---

Table 4: Results of using historical domain distributions in different ways during adversarial training
Yearbook
RENNSST
IMOW
Hutipoe
Artiv
Used historical domain diseribution Cuss
Accuracy (%)
Accuracy (%)
Accuracy (56)
Accuracy (5))
Accuracy (5))
Method
D144
O00
000
D3441
OOD
000
DD-41
0OD
ODD
DF+1
0OD
0OD
1F+1
DOD
ODD
aeg.
w0xst
woost
Wosst
avg.
wosst
avg.
Wosst
Exn5 Evo8
rendmm
95.05
93.04
85.46
66.62
46.76
72.00
55.80
52.45
historical
95.53
93.84
67.18
73442
72.36
70.19
53.15
Eva6 denotes that nandonaly solect domann dieriuuii from the mensary pool A sample buih of Scatono for panticipeiig the advemaria traniing.
Mean
Variance
100.0
0
Durs
Ours Compared
OCD avg. accurary (%g
0.7
Compared
97.5
000 socuraoy
97.5
0.6
0.4
0.5
fx 1o-norm
95.0
0.3
92.5
92.5
17
17.3
13
1
1)-2
13
13
17
5 6 6,
Domain
Domein
Number of heads P.
Valuo scale
(a) Discrepancy between real and generated statistics.
(b) Effect of the number of heads and scale.
Figure 2: (a): The 1-2 norm of the difference A between the statistics calculated on each test domain and the statistics generated by MSAM on Yearbook. "Ours" refers to the way in Eq. an and "Compared" refers to minimizing the 1-2 norm of A between statistics calculated in each batch and that generated by MSAM. (b): Effects of using different numbers of heads and scales on Yearbook.
variant D, E and F are all inferior to EvoS. Such results verify two points. One is that the adversarial training conduces to better generalization and mitigating the problem of overfitting. And the other is that the consistency between generated statistic tokens in the window and real statistic tokens in the memory pool imposes stronger constraint to attention module A.. helping to model the evolving pattern more accurately. Thirdly, we testify the influence of truncation when sampling features from normal distributions. The worse result of variant G than EvoS may be due to the sampled outliers hinder the learning of shared feature space. Finally, the results of variant H and EvoS compare single-scale and multi-scale attention. The better result of EvoS shows the superiority of MSAM.
Last but not least, in Table we investigate the effect of randomly selecting a historical domain distribution vs using all historical domain distributions in the adversarial training. From the results, we see that randomly selecting one historical domain distribution performs worse. This may be because the feature space to be aligned frequently changes if using this manner, making the optimization challenging. By contrast, it is more stable to simultaneously leverage all the preserved domain distributions the memory pool M in each iteration for the adversarial training.
Choices of Cestnn. In addition to training Aw via loss Lasan in Eq. 10 we also try another way. Concretely, we directly minimize the (-2 norm of the difference between statisties calculated in each batch and that generated by MSAM, i.e., Ca = 1 Σ.e sg( p I|2 + E' Σ (sg(
1 >ー sf(f)) ( 12. Fig. 2(a) plots the discrepancy between generated and real statistics B on each test domain of Yearbook when using the two ways to train the attention module. We see that the way used in Sec. B3 yields smaller discrepancy between generated and real statistics, implying that the learned evolving pattern is more accurate. So we choose to use it throughout the experiments.
Effect of Number of Heads and Scale. We further investigate the effect of number of heads and scale of MSAM in Fig. 2 Varying the number of heads, the performance exhibits an inverted V-shaped trend, because too few heads would hamper the learning capability of the attention module while too many heads could lead to overfitting. As for difference scales (i.e., the maximum length of sliding time window), we find that multi-scales (W 2 1) yields better performance than single-scale (W = 1), showing the superiority of MSAM. Yet, more scales do not always produce better results. One possible explanation is that an excessively large scale relative to the domain sequence length leads to an inadequate number of sliding windows, hampering the learning of attention module.
Visualization of Decision Boundary. In this qualitative experiment, we visualize the change of decison boundary on the inter-twinning moons 2D problem in a gradual manner. Concretely, the 2-Moons dataset in [41] is used, where 10 domains (0 to 9) are obtained by rotating data points counter-clockwise in units of 18 In Figure the model is sequentially trained using EvoS until the (-th domain is finished, and then we visualize the decision boundary next future domain D4+1. From the results, we can observe that the decision boundary successfully adapts to future domains, showing that EvoS can truly capture the underlying temporal drift of data.
current domain D' and the

---

y'y== 1y +1 y=t1,y=-1 y++
(a)
(c) t=
(d) t=
Figure 3: Visualization of decision boundaries for the model at current time point on inter-twinning moons 2D problem. Blue line and points are the decision boundary and samples current domain D', and red line and points are the decision boundary and samples in the next future domain D4+1
OOD avg. accuracy (%)
95
100
RD
90
80
70
60
50
000
15.0
G.h5 52,2255 0.1
10.0
3 16
5.0
Yearbook
Lengtn memory
(a) t-SNE visualization of test domains.
(b) Effect of pool length L. (c) Sensitivity of a A.
Figure 4: (a): Visualization of standardized features from each test domain Yearbook and (b): Effect of the memory pool length L on dataset Yearbook. SaHHyper-paramete sensitivity of and À Yearbook
RMNIST. Different colors denote different domains.
Effeet of Memory Pool Length. In the main experiments, we do not restrict the size of the memory pool M, since the datasets used in paper has a moderate number of domains and the memory cost small. Nevertheless, a fixed memory pool size is more practical when considering a lifelong process, i.e., T - c0. Thus, additionally conduct the experiment where the memory pool M is implemented as a FIFO queue with different fixed length L. That is, only the statistics for up to the most recent historical domains be stored. The results Yearbook in Fig. S(m)onssttttrt that our method generally performs well and a relatively large memory pool length is better.
t-SNE Visualization of Standardized Features. To verify our method appropriate for generalization, we visualize the standardized features of all target domains via t-SNE [38] on RMNIST and Yearbook in Fig. 4{a) It can be observed that the standardized features of target domains are well aligned, suggesting that MASM can capture evolving patterns effectively and feature standardization helps address temporal drift appropriately. These enable EvoS to achieve generalization future domains.
Hyper-parameter Sensitivity. a and control the truncation range and the tradeoff the adversarial loss Lade. respectively. Fig. [(c)sshws the sensitivity of EvoS to them on Yearbook, where a (0.1, 0.5, 1.0, 1.5, 2.0, 2.5. 3.0} and E (0.1, 0.5, 1.0, 1.5, 2.0, 5.0, 10.0, 15.0, 20.03. We see that EvoS is more sensitive to than and large À values significantly worsen performance. In practice, we suggest selecting from a range l. For a, its effect is modest and = generally works well.
Conclusion
This paper considers drift (CDGTD) than conventional DG, where the model is incrementally trained with sequential domains and is required to generalize to unseen domains that are not too far into the future. To this end, we propose an Evolving Standardization (EvoS) method to learn the evolving pattern of sequential domains over the temporal drift and hope to achieve the generalization by transforming the domain distribution into a common normal distribution via feature standardization. Experiments on real-world datasets including images and texts verify the efficacy of EvoS. Since existing DG works focus on conventional setting, we hope this work can encourage more research works on CDGTD.
challenging problem of continual domain generalization over temporal
10

---

Acknowledgements
This paper was supported by National Key R&D Program of China (No. 2021 the National Natural Science Foundation of China (No. 62376026), and also sponsored by Beijing Nova Program (No. 20230484296).
References
[1] R. Aljundi, M. Lin, B. Goujaud, and Y. Bengio. Gradient based sample selection for online continual learning. In NeurlPS, pages 11816-11825, 2019.
[2] M. Arjovsky, Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv: 1907.0289, 2019.
[3] G. Bai, C. Ling, and L. Zhao. Temporal domain generalization with drift-aware dynamic neural networks. ICLR, 2023.
[4] Ben-David, Blitzer, K. Crammer, A. Kulesza, Pereira, and J. W. Vaughan. theory of learning from different domains. Mach. Learn., 79(1-2)1151-175, 2010.
[5] S. Ben-David, J. Blitzer, K. Crammer, and Pereira, Analysis of representations for domain adaptation. In NeurIPS, pages 137-144, 2006.
[6] G. Blanchard, G. Lee, and C. Scott. Generalizing from several related classification tasks to new unlabeled sample. In NeurlPS, pages 2178-2186, 2011.
[7] G. Blanchard, G. Lee, and C. Scott. Generalizing from several related classification tasks to a new unlabeled sample, 2011.
[8] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
[9] M. Castro, M. J. Marín-Jiménez, N. Guil, C. Schmid, and K. Alahari. End-to-end incremental learning. In ECCV, pages 233-248, 2018.
[10] A. Chaudhry, M. Ranzato, M. Rohrbach, and M. Elhoseiny. Efficient lifelong learning with A-GEM. In ICLR, 2019.
[11] T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton. A simple framework for contrastive learning of visual representations. In /CML, pages 1597-1607, 2020.
[12] X. Chen, W. Chen, V. Dinavahi, Y. Liu, and J. Feng. Short-term load forecasting and associated weather variables prediction using resnet-lstm based deep learning. JEEE Access, 11:5393- 5405, 2023.
[13] L. Deng. The MNIST database of handwritten digit images for machine learning research [best of the web]. SPM, 2966):141-14,, 2012.
[14] S. Ebrahimi, F. Meier, R. Calandra, T. Darrell, and M. Rohrbach. Adversarial continual learning. In ECCV, volume 12356, pages 386-402, 2020.
[15] Y. Ganin and V. Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML, pages 1180-1189, 2015.
[16] S. Ginosar, K. Rakelly, S. Sachs, B. Yin, and A. A. Efros. A century of portraits: A visual historical record of american high school yearbooks. In ICCV Workshops, pages 652-658, 2015.
[17] R. Gong, W. Li, Y. Chen, and L. Van Gool. Dlow: Domain flow for adaptation and generalization. In CVPR, 2020.
[18] G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. CVPR, pages 2261-2269, 2017.
[19] Y. Iwasawa and Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. In NeurIPS, pages 2427-2440, 2021.

---

[20] Y. Iwasawa and Y. Matsuo. Test-time classifier adjustment module for model- domain generalization. In NeurlPS, pages 2427-2440, 2021.
[21] P. Izmailov, D. Podoprikhin, T. Garipov, D. P. Vetrov, and A. G. Wilson. Averaging weights leads to wider optima and better generalization. In UAI, pages 876-885, 2018.
[22] J. Kirkpatrick, R. Pascanu, N. C. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan,
J. Quan, T. Ramalho, A. Grabska-Barwinske, D. Hassabis, C. Clopath, D. Kumaran, and
R. Hadsell. Overcoming catastrophic forgetting in neural networks. arXiv:1112.0099, 2016.
[23] A. Kumar, T. Ma, and P. Liang. Understanding self-training for gradual domain adaptation. In ICML, pages 5468 5479, 2020.
[24] M. D. Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. G. Slabaugh, and T. Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. TPAMI, 44(7):3366-3385, 2022.
[25] D. Li. Y. Yang. Y. Song, and T. M. Hospedales. Deeper, broader and artier domain generalization. In ICCV, pages 5543-5551, 2017.
[26] D. Li, Y. Yang. Y. Song, and T. M. Hospedales. Learning to generalize: Meta-learning for domain generalization. AAAI, pages 3490-3497, 2018.
[27] H. Li, S. J. Pan, S. Wang, and A. C. Kot. Domain generalization with adversarial feature learning. In CVPR, pages 5400-5409, 2018.
[28] Li, C. Liu, Q. Lin, Q. Wen, Su, G. Huang, and 7. Ding. Deep residual correction network for partial domain adaptation. TPAMI, 43(7):2329-233 2021.
[29] S. Li, C. H. Liu, B. Xie, L. Su, Z. Ding, and G. Huang. Joint adversarial domain adaptation. In ACM MM, pages 729-737,2011
[30] Z. Li and D. Hoiem. Learning without forgetting. TPAMI, (0(12):2935-2944) 2018
[3]] Lin, Y. Wang, X. Liu, and X. Qiu. survey of transformers. Al Open, 3:1 2022.
[32] A. Liu, Y.C. Liu, Y.-Y. Yeh, and Y.-C. Wang. unified feature disentangler for multi-domain image translation and manipulation, 2018.
[33] H. Liu, M. Long, J. Wang, and Y. Wang. Learning to adapt to evolving domains, 2020.
[34] M. Long, Z. Cao, J. Wang, and M. I. Jordan. Conditional adversarial domain adaptation. In NeurlPS, pages 1647-1657, 2018.
[35] Lv. J. Liang, K. Gong. S. Li, C. H. Liu, H. Li, D. Liu, and G. Wang. Pareto domain adaptation. In NeurlPS. pages 12917-12929, 2021.
[36] Lv, J. Liang, S. Li, B. Zang, H. Liu, Z. Wang, and Liu. Causality inspired representation learning for domain generalization. In CVPR, pages 0046-8046, 2022.
[37] Lv, J. Liang, S. Li, Zhang, and D. Liu. Improving generalization with domain convex game. In CVPR, pages 24315-24324, 2023.
[38] L. v. d. Maaten and G. Hinton. Visualizing data using t-sne. JMLR, 9:2579-2605.. 2008. 2008.
[39] K. Muandet, D. Balduzzi, and B. Schölkopf. Domain generalization via invariant feature representation. In ICML, pages 10-18, 2013.
[40] Z. Nado, S. Padhy, D. Sculley, A. A'mor,r, B. Lakshminarayanan, and J. Snock. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv:2006. 10963, 2020.
[41] A. Nasery, Thakur, Piratla, A. De, and S. Sarawagi. Training for the future: simple gradient interpolation loss to generalize along time. In NeurIPS, pages 19198-19209, 2021.
12

---

[42] A. Nguyen. T. Tran, Y. Gal, and A. Baydin. Domain invariant representation learning with domain density transformations, 2021.
[43] G. Ortiz-Jiménez, M. E. Gheche, E. Simou, H. P. Maretic, and P. Frossard. CDOT: continuous domain adaptation using optimal transport. arXiv: 1448, 2019.
[44] T. Qin, S. Wang, and H. Li. Generalizing to evolving domains with latent structure-aware sequential autoencoder. In ICML, pages 18082, 2022.
[45] J. Rajasegaran, M. Hayat, S. H. Khan, F. S. Khan, and L.. Shao. Random path selection for incremental learning. 10Xiv:006.0112200 2019.
[46] S. Rebuffi, A. Kolesnikov. Sperl, and H. Lampert. icarl: Incremental classifier and representation learning. In CVPR, pages 5533-5542, 2017.
[47] G. Saha and K. Roy. Continual learning with scaled gradient projection. In AAAI, pages 9677-9685, 2023.
[48] V. Sanh, L. Debut, J. Chaumond, and Wolf. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv: 2019.
[49] S. Schneider, F. Rusak, I.. Eck, O. Bringmann, W. Brendel, and M. Bethge. Improving robustness against common corruptions by covariate shift adaptation. In NeurIPS, 2020.
[50] B. Sun and K. Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV Workshops, pages 443-450, 2016.
[51] R. Taori, A. Dave, V. Shankar, N. Carlini, B. Recht, and L. Schmidt. Measuring robustness to natural distribution shifts in image classification. In NeurIPS, 2020.
[52] R. Tiwari, K. Killamsetty, R. K. Iyer, and P. Shenoy. GCR: gradient coreset based replay buffer selection for continual learning. In CVPR, pages 99-108, 2022.
[53] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation the real world. In IROS, 2017.
[54] D. Wang. E. Shelhamer, Liu. B. Olshausen, and T. Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021.
[55] H. Wang, H. He, and D. Katabi. Continuously indexed domain adaptation. In ICML, volume 119, pages 9898-9907, 2020.
[56] H. Yao, C. Choi, B. Cao, Y. Lee, P. W. Koh, and C. Finn. Wild-time: benchmark of in-the-wild distribution shift over time. In NeurlPS, 2022.
[57] H. Yao, Y. Wang, S. Li, I.. Zhang, W. Liang, Zou, and C. Finn. Improving out-o--distributiin robustness via selective augmentation. In ICML, pages 25407-25437, 2022.
[58] L. Yuan, B. Xie, and S. Li. Robust test-time adaptation in dynamic scenarios. In CVPR, pages 15932, 2023.
[59] Q Zeng, Wang, Zhou, C. Ling, and B. Wang. Foresee what you will learn: Data augmentation for domain generalization in non-stationary environments. arXiv:2301.07845, 2023.
[60] Zenke, B. Poole, and S. Ganguli. Continual learning through synaptic intelligence. In ICML, pages 3987-3995, 2017.
[61] H. Zhang, M. Cissé, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk mini- mization. In ICLR, 2018.
[62] K. Zhou, Z. Liu, Y. Qiao, T. Xiang, and C. C. Loy. Domain generalization: A survey. TPAMI, 45(4):4396-444 4415, 2023.
[63] K. Zhou, Y. Yang, Qiao, and T. Xiang. Domain generalization with mixstyle. In ICLR, 2021.
13

---

Appendix Contents
A. Broader Impacts & Limitations
B. Ethies Statement & Licenses
C. Algorithm of EvoS
D. Experimental Sctup Details
E. Experimental Results with Error Bars
F. Complexity Analysis
G. Additional Results
A Broader Impacts & Limitations
Our work focus on the problem of continual domain generalization over temporal drift (CDGTD), which aims to generalize the model to unseen future domains by leveraging underlying evolutionary patterns. The effectiveness of our method on several real-world datasets means that it may potentially benefit relevant applications and communities that deal with temporal drifts, e-g., advertisement recommendation, autonomous driving, popularity forecast of media content, etc. Nevertheless, we should also be cautious about possible failures of our method when encountering sudden distribution shifts. In the future, we may explore the automatic identification of domains with severe distribution shifts, allowing to proactively reject them and mitigate the risk of severe accidents.
B Ethics Statement & Licenses
All the datasets used in the paper are publicly available and are only intended to compare the performances different algorithms on classification tasks, adhering to the following licenses of these datasets:
Yearbook: MIT licensed
MNIST: BY-SA 3.0
fMoW: The Functional Map of the World Challenge Public License
Huffpost: CCO: Public Domain
arXiv: CCO: Public Domain
C Algorithm of EvoS
The training and inference procedure for EvoS is provided Algorithm and respectively.
D Experimental Setup Details
D.1 Dataset Description
Yearbook used in our paper is from [66] It is based on the Portraits [16] dataset (MIT license), containing the 32 x 32 grayscale images of yearbook portraits from 128 American high schools in 27 states. The data spanning from 1930 to 2013 reflects the evolving fashion styles and shifting social norms throughout the decades. The task is the binary classification of genders. Note that it is only used for comparing the generalization performances of different algorithms on classification tasks. For this dataset, we treat every four years domain, resulting in 16 domains. Table [1 provides the number of samples in each domain. The first 16 domains are used for training (T 16), the last domains for testing (K- 5).
RMNIST is a variant of the MNIST the rotations with degree of 09, 10" ... 80 in order to MNIST to simulate the temporal drift. The task classify 28 28 grayscale digit image from We use the first 6 domains for training (T 6) and the last 3 for testing (K 3).
dataset, which comprises 9 domains generated by applying
14

---

Algorithm 1: Training procedure for EvoS
Input:
sequential labeled training domains (D), "D,..., , ,'T,, hyper-parameter a, X, W, feature encoder &, classifier C, learnable statistic vectors fi aa. f a", multi-head self attention modules {Aw J memory pool M, batch size B, training steps E.
for / t= to do
train on t-th domain
for e to E' do
Randomly sample a batch of data (æf, y/ h from domain D
Obtain features (fie by feeding {0)) into the feature encoder & Ciotall = 0. 18
multi-scale attention module
for w = to W do if 2 w 2 then
Prepare the input S- = an s _ for A, by using sliding time
10
window with length w on the memory pool M, as described Eq. 4.
Obtain the output 5 (S
[ S :..... (
Compute the average of output tokens: st 1d-w+1 avg( w
2i=1
8-1
13
Split uo Calculate loss L by Eq. 11 and Ciotal = C
s--w+1 into w parts: ( [st-w+1, s/y 8t x w+1
14
15
calculate losses
if >==3>> 3 then
17
Generate the statisties for current domain [A =')=="" 8¹
W - 8
18
Caleulate loss Lte by Eq. 8 and Lyton by Eq. 10 and L (Cte + L. ssaan).
19
i 2 2 then
Sample features [ BB
t-1 by Eq. 12 as the proxy of historical domains. Calculate loss L'ddo by Eq. 13 and =
20
21
22
Optimize corresponding modules by minimizing loss Ci
23
store statistics into the memory pool M
4/
24
it' >= 3 then
25
Generate statistics fi', â' by repeating step 8 to 17.
Store the learned/generated statistics into the memory pool M by u a + aa'
26
return Final & C, (Aw [..)
27
IMoW used in our paper is from [56]. It collects the 224 x 224 RGB satellite images from 2002 to 2017 over 200 countries, where the visual features present in satellite data undergo changes over time due to both human and environmental activities. The task to classify the functional purpose of the buildings or the land the images into one of 62 categories. For this dataset, we treat every year as a domain and Table 12 provides the number of samples in each domain. The first 13 domains used for training (T 13) and the last 3 domains are used for testing 333))
Huffpost in [66 comprises news headlines from the Huffington Post from 2012 to 2018, the task of which is to classify the news headline into one of 11 news categories ("Black Voices". "Business", "Comedy". "Crime", "Entertainment" "Impact", "Queer Voices", "Science", "Sports", "Tech", "Travel"). The data spanning for 2012 to 2018 presents changes the content or style of news along the time dimension. For this dataset, use the first years for training (I 4) and the last 3 years for testing (K = 3). The number of samples each domain is provided in Table [3
Arxiv in [56] contains paper titles and their corresponding primary categories spanning from 2007 to 2022. The content of Arxiv preprints evolves over time, reflecting the dynamic nature of research fields. And the task is to classify a research paper into one of 172 categories based on its title. We use the data from the first domains for training (T = 9) and the data from the last years for testing (K 7). The number of samples in each domain is presented in Table [4
15

---

Algorithm 2: Inference procedure for EvoS
Input: sequential target domains (DT+1, DT+2 ... ,DT+K), feature encoder &, classifier C, ii multi-head self attention modules (An J w=1- memory pool M.
for == T + to T + K do
/
generale statistics via multi-scale attention module
+/
3
for w =-=---1 to W do
Prepare the input 5-11 = w w .... : "] for Aw by using a sliding time window with length on the memory pool M, as described in Eq. 4.
Obtain the output $ Aw (5" m = [ wi: ... 3t-w] w
Compute the average of output tokens: s x--++1 avg(S 5 10 1-
EiSr
Split x into we parts: [$t-=+1
s [ty1 x-- --+1
Generate the statistics for current domain as pp a'I = W
Ew
10
inference on 1-th domain
10
for xx E D do
Obtain features f by feeding x' into the feature encoder E.
11
Conduct feature standardization by z' s f'
f'-p' exp(a"))
13
Generate the prediction ÿ' by feeding z' into the classifier C.
14
store statistics into the memory pool M
#/
15
Store the generated statistics into the memory pool M by μ², a' t p',aaa a'.
For each training domain all datasets, we randomly select 90% of data as training split and 10% of data as validation split. The evaluation of generalization performance is based on the whole data of each test domain.
D.2 Implementation Details
All experiments are implemented via PyTorch and the backbones we use mainly adhere to [56]. Yearbook uses 4-layer convolutional network from (56) as the backbone. For multi-scale attention module (MSAM), we set the dimension of head as d,, = 8 for each attention module A. and the number of heads for A, as w Th, with ni = 16, w = 1, .... W. And the Adam optimizer with Ir = le 3 is used to optimize the model, where the batch size B is set to 64 and the training epochs of each domain are set
As for the hyper-parameters, we use a = 1.0, X 1.0, W
RMNIST adopts the ConvNet in [44] as the backbone, and we set the dimension of head as d, = 8 for each attention module Aw and the number heads for An as w Th, with n4 32, w 1.... W. The Adam optimizer with Ir = le - 3 is adopted for model optimization. The batch size and training epochs of each domain are set to 64 and 50, respectively. And a 2.0, X 1.0, W 3 is used.
IMoO employ the DenseNet- 121 18 pretrained on ImageNet as the backbone. Besides, we bottleneck layer (28 reduce the feature dimensions into 256, and set the dimension of head as dy 8 for each attention module An and the number of heads for A,p as w nh, with n4 64, 1r=22 = 2e 4. The batch size is set to = 64 and each training domain is trained for 25 epochs. As for the hyper-parameters, we set a 1.0, 1.0, W 3 for iMoW dataset.
1, ... W. Similarly, the Adam optimizer is used, where the learning rate is set to
Huffpost and Arxiv use the pretrained DistilBERT base model 48 as the backbone. For MSAM, we set the dimension of head as d, 8 for each attention module A., and the number of heads for Aw as w nh, with n, = 128, w = 1. ..... W. Also the Adam optimizer is used with the learning rate Ir = 2e 5 and batch size B 64. And a 1.0. À 1.0, W is used for the two dataset. For the Huffpost dataset, each training is trained for 50 epochs, while the training epochs of each domain are to 5 for the Arxiv dataset.
We run each task on a single NVIDIA GeForce RTX 3090 GPU for three random trials. For baselines, we also select their hyper-parameterd via the grid search using the validation splits of training domains.
16

---

Table 5: Accuracy (%) on Yearbook, RMNIST and fMoW. The best and second-best results in
CDGTD setup are bolded and underlined. (Yearbook: K 5, RMNIST:: K 3, fMoW: K 3)
Method
Inceesental
domains
muhiple
Access
Accurnty (93
Yeutlook
O010
D00
12
Ascuracy
RMNNST
030
000
17
word
Gillinc
10
88.48
12.723
36.81 (3100)
55.
10.41)
922
14 (0:91)
83.80 35)
12.43 (1.31)
4085 (500
IKM
97.09 (0.35
94.52
(0.41)
92.55
(0.522
95. 10. 18.
85.05 (3.17)
72.52
13.51
64.77 (1.57)
54.92 (2.00)
46.51 (3:47)
DOKAL
NC
95. 29.59 (1.433
91.29
(3.44)
89.86 16.713
95.05 60. 553
79.10
(1 (.50
63.95
11:00
62.14 (1.32)
51.42 (2.15)
42. (3.00)
Miuu 61
94.98 1949
91:12 (1 073
88. 55 (3.45)
97.11 00.413
89.66 (0:09)
79 03
11 472
70.27
(0.91)
57-21 (1:15)
48.044) (14)
LISA 591
95.51 (0.75)
92.97
(0.57)
91.29
(0.42)
96.21 (0.51
87.04 (1.94)
75.15 12.43
70005 (0.54)
55.52 (0:56)
44,61 (1.02)
CDOT 45
95.07 1.955
92.90
90.67 (1 (1.81)
91.45
12.
97.96 (3.00)
91.19 (0.93)
79:07
11 271
CIDA
92.36 225
45 099
97.45 (80
89.
(0.29)
78. 52
461
(3.32)
96.
(0.38)
97.78
(0.70)
12.011
61.62 (1.27)
(2.19)
(2.57)
LSSAE
(1.91)
92.12
90.36 (0.46)
82.
(1:50)
41:38
IncFaeeune
96.81 (0L17)
94.72
98.62
92.80 (0.65)
84.63
65.52
(0.46)
45:23 (0.85)
Miupp
67)
(0:75)
66.58 84
43.54
SimCLK 11
(3.35)
93007 (0.67)
(0.
90.95 (0:61)
81.01
(0.59)
44.71 (0,53)
(0.12)
94.27 (0.07)
91.44
90.35 (0.38)
10:351
(0.18)
45.29 (0:11)
95.12 (0.07)
92.02 (0:31)
(0.10)
(0.
(1:21
(0.11)
A-GEM :0
(3.09)
90.96 (0.51)
(0.29)
95.99 (3.66)
75.45
54.54 (1.98)
(2.46)
41.13 (3.57)
93.00
67.22
55.05
46.24 12124
94.78
(0.16)
The IMW: haddhore DiresNNes lonno brg to rpp full and DRAIN So Appl TIRAIN only to clandltr and apply C2 withoon the tee-iiiiigg ergn.
Table 6: Accuracy (%) on Huffpost and Arxiv. The best and second-best results in CDGTD setup are
bolded and underlined. (Huffpost: K 3. Axriv: K = 7)
Method
Incremental training
multiple domains
Access
DT=1
Accuracy (
Huitpost
ODD
OOD
DT++1
Arxiv Accuracy (S)
0OD
ODD
avg.
worst
Offine
72.74 (0.14)
71.50 (0.56)
69.63 (0.81)
57.49 (0.15)
52.38 (0.43)
49.28 (0.67)
IRM 2
V
71.04 (0.45)
70.31 (0.67)
68.97 (0.87)
$1.11 (1.04)
45.89 (2.77)
42,86 (3.98)
CORAL 50
70.08
(0.94)
50.98
(4.15)
Mixup 61
V
73.34 (0.02)
71.16 (0.07)
69.29 (0.12)
57.58 (0.03)
$2.77 (0.19)
49.62 (0.24)
LISA 57
V
72.19 (0.06)
70.24 (0.56)
68.60 (0.91)
56.53 (0.02)
52.41 (0.11)
49.67 (0.23)
[4]
68.06 (1.51)
66.32
(3.67)
53.43 (1.65)
(2.49)
[nc] Finetune
X
73.37 (0.02)
71.98 (0.06)
69.80 (0.11)
$6.22 (0.02)
52.43 (0.14)
49.37 (0:21)
Mixup [1
73.07
71.52
(0.29)
56.64
52.95 (0.06)
49.97 (0.11)
EWC 22
X
73.64 (0.02)
71.53 (0.29)
68.99 (0.46)
$6.60 (0.04)
52.78 (0.07)
49.73 (0.12)
(0
72.58
(0.34)
49.98
47.27 (2.46)
44.77
A-GEM [0
X
72.23 (0.06)
71.16 (0.14)
69.10 (0.28)
52.02 (0.99)
48.91 (2.67)
46.03 (2.09)
DRAIN 3
X
73.42 (0.02)
71.75 (0.16)
69.69 (0.20)
56.04 (0.06)
$2.07 (0.18)
48.97 (0:34)
Evo8
X
73.42 (0.02)
72.36 (0.05)
70.19 (0.08)
56 (0.02)
53.15 (0.04)
50.19 (0.10)
For [lulppstt and Arais, backbone DisciBERTT buse lou big apply the full and DRAIN, So we apply DRAIN only to the classifier and upply wilhout the line- line-tuning stage.
E Experimental Results with Error Bars
In this section, we report the mean and standard derivation (denoted as mean (std) ) for each task, when running with 3 random trials. The results with error bars on Yearbook, RMNIST and fMoW are provided Table and the error bars on Huffpost and Arxiv are given in Table 6
F Complexity Analysis
Time Complexity of MSAM. Taking one of the multi-head self-attention module Air in MSAM as an example. we denote d, as the feature dimension of its input tokens, and da, niş and n. denote the feature dimension of its heads, the number of its heads and the number of its input tokens, respectively. Then the time complexity of A., is (( + ni dy) (dr- - (n))). Since n, will be no larger than the number of training domains T and dy n4 is usually set to d in transformers, the time complexity of MSAM can be roughly approximated as O(W (T²d, + T de)), where W is the number of multi-head self-attention modules in MSAM. It is roughly equivalent to the time complexity of single multi-head self-attention layer in conventional transformers multiplied by W.
Memory Complexity of M. Assuming that there are T historical domains and the dimension of statistic vectors is d,, then the memory complexity of the memory pool M is OT -dy) In practice, after being processed by deep neural networks, the dimension of pooled features is usually much smaller than that of original inputs. Moreover, only two vectors need to be stored per domain. Hence, the memory cost of M is relatively small, compared with sample replay-based CL methods. Concretely, Table provides the memory cost of M and the increment of GPU memory cost for EvoS on Yearbook, RMNIST and fMoW datasets with batch size 64.
17

---

Table 7: Memory cost on Yearbook, RMNIST and fMoW datasets.
GPU memory cost (GB) different methods
Dataset
Yearbook
RMNIST
iMoW
Method
Backbone
4-layer CNN [56]
ConvNet [44]
DenseNet-IP 121 [18]
IncFinetune
1.72
1.84
10.69
EvoS
1.94
2.09
11.04
Increment (GB)
0.22
0.25
0.35
Memory cost (MB) of the memory pool M
EvoS
5.25
9.00
32.00
Table 8: Model parameters (MB) of different methods Yearbook, RMNIST and fMoW.
Dataset
Yearbook
RMNIST
tMoW
Method
Backbone
4-yyyer CNN [56
ConvNet [44]
DenseNet-12 121 [18]
LSSAE [44]
4.70
23.25
90.92
DRAIN EvoS
7.51
184.29
1113.85
1.94
15.81
56.11
Model Complexity. Here, we measure the model complexity by the number of parameters. Specifi- cally, Table presents the model complexity of our method EvoS and other two temporal DG methods LSSAE [44] and DRAIN [3] Yearbook, RMNIST and fMoW datasets. LSSAE 44) introduces sequential autoencoder to explore the underlying continuous structure in the latent space of deep neural networks, where the complicated VAE and LSTM networks require lots of parameters. And DRAIN 3 needs to encode and decode the entire network parameters, which also requires a great number of parameters for large backbone networks. By contrast, our EvoS is overall less complex than these temporal DG methods, and is more friendly to the relatively large backbone network.
G Additional Results
Results under Eval-Stream Manner. In this part, additionally provide the results when adopting the evaluation manner of Eval-Stream in [56]. Specifically, Eval-Stream denotes the evaluation with domain stream, i.e., the model is evaluated at each timestamp using the average and worst performance on the next K timestamps. Formally, given sequence of domains with total length T, the average performance Augstream and worst performance W Worstatream under the strategy of Eval-Stream are defined as Augstream = T-K -T-K DitK (Dc,(D)), W Worststream K
-LR ΣT-K min .) (Do),D)), where (D); is the accuracy on domain D when using the model at the i-th timestamp. And Table 9 shows the results on Yearbook and Huffpost datasets when using the Eval-Stream evaluation manner. According to the results, we see that EvoS still outperforms other baselines, showing that our method better handling the problem of continual domain generalization over temporal drift.
Table 9: Accuracy (%) Yearbook and Huffpostt under the evaluation manner of Eval-Stream.
(Yearbook: T 21, K 5, Huffpost: T = 7,
3)
Yearbook
Huffpost Accuracy (%)
Accuracy (%)
Method
Conference
Augstream
Worststream
Augstream
W or Ststremm
IncFinetune
89,67
83.56
67.97
64.11
Mixup 61
ICLR' 18
84.79
78.69
67.33
63.48
SimCLR 11
ICML''0
89.50
83.16
SWAV 8
NeurlPS 20
90.05
84.08
EWC 22
arXiv'16 16
90.15
83.75
68.31
64.58
SI [60
ICML'17
90.14
84.07
67.93
64.05
A-GEM [10
ICLR'19 19
84.40
77.59
65.83
62.20
DRAIN 3
ICLR'23
87.26
81.95
68,04
64.14
EvoS
90.43
84.48
68.66
64.97
18

---

Table 10: Misclassification error (%) on 2-Moons, ONP and Elec2. (K' = 1)
Misclassification error (in %) |
Method
Conference
2-Moons
ONP
Elec2
Offline
22.4+4.6
33.8=0.6
23.0=3..1
LastDomain
14.9=0.9
36.0#0.2
25.8+0.6
IncFinetune
16.7+334
34.0+10.3
27.3+44.2
CDOT 43
arXiv' 19
9.3+=1.0
34.1+0.0
17.8++0.6
CIDA 55
1CML:20
10.8+1166
34.7+-0.6
14.1+0.2
GI 41
NeurIPS'21
3.5+14
36.420.8
16.9
DRAIN 3
ICLR'24
3.2+122
38.3+112
12.7+008
EvoS
2.5
33.1
11.610.7
Results on More Datasets. In addition to the datasets provided in [56]. also run method on the 2-Moous, ONP and Elec? datasets used in [3] Specifically, 2-Moons is a variant of the 2-entangled moons dataset by rotating data counter-clockwise in units of 18 to construct 10 domains, where the rotation angle is used to simulate the temporal shift. For 2-Moons, we use Adam optimizer and MLP with two hidden layers of hidden size 64 and 128, and B 64, Ir le 3, 1.0, X 1.0, W = 3, dh = 8, n' = 32, T 9, K 1. Online News Popularity (ONP) summarizes heterogeneous set of features about articles published by Mashable in a period of two years. The dataset is split into 6 domains by time and the goal is to predict the number of shares in social networks (popularity). For ONP, we use Adam optimizer and MLP with one hidden layer of hidden size 128, and B = 64. Ir = le 4, = 1.0, À 1.0, W = 3, dn = 8, nu 32, T 5, K 1. Electrical Demand (Elec2) contains information about the demand of electricity in a particular province. Following 3 41 the first 30 domains in Elec2 are used (two weeks as one time domain) and the task is to predict whether the demand electricity in each period (of 30 mins) higher or lower than the average demand over the last day. For Elec2, we use Adam optimizer and a MLP with two hidden layers of hidden size 128 and 128, and = 64, Ir
le 4, a 1.0, À 1.0, W da 8, 71A 32, T 29, K 1. For these datasets, as in 3 41l. use the last domain for testing and the rest for training. Please refer to for more dataset details. The experimental results given in Table TO where the misclassification errors of compared baselines are all reported from DRAIN B.. From Table 10 we can observe that our method EvoS still surpasses the most recent method DRAIN, affirming its effectiveness in temporal domain generalization.
Table 11: Data Subset Size for the Yearbook Dataset.
Domain
Interval
Training Split
Validation Split
All
1930 1933
758
87
845
1934 1937
1149
130
1279
1938 1941
949
108
1057
1942 1945
2353
263
2616
1946 1949
1229
138
1367
1950 1953
1082
122
1204
1954 1957
1646
185
1831
1958 1961
1295
146
1441
1962 1965
1468
166
1634
1966 1969
2227
249
2476
11
1970 1973
1634
183
1817
1974 1977
2238
250
2488
13
1978 1981
1553
175
1728
1982 1985
2331
261
2592
15
1986 1989
1792
201
1993
1990 1993
1729
195
1924
17
1994 1997
1882
211
2093
18
1998 2001
2136
239
2375
19
2002 2005
1868
210
2078
20
2006
2009
1010
114
1124
21
2013
1102
125
1227
total
1930 2013
33431
3758
37189
19

---

Table 12: Data Subset Size for the fMoW Dataset.
Domain
Year
Training Split Validation Split
All
2002
1676
227
1903
2
2003
2279
276
2555
2004
1755
240
1995
4
2005
2512
324
2836
2006
3155
406
3561
2007
1497
190
1687
2008
2261
298
2559
8
2009
7439
935
8374
2010
18957
2456
21413
10
2011
22111
2837
24948
2012
24704
3138
27842
2013
3465
385
3850
13
2014
5572
620
6192
2015
8885
988
9873
15
2016
14363
1596
15959
2017
5534
615
6149
total
2002-2017
126165
15531
141696
Table 13:
Data Subset Size for the Huffpost Dataset.
Domain
Year
Training Split Validation Split
All
2012
6701
744
7446
2
2013
7492
832
8325
3
2014
9539
1059
10599
2015
11826
1313
13140
5
2016
10548
1172
11721
6
2017
7907
878
8786
7
2018
3501
388
3890
total
2012-2018
57514
6386
63907
Table 14: Data Subset Size for the Arxiv Dataset.
Domain
Year
Training Split
Validation Split
All
2007
131550
14616
146167
2
2008
62460
6939
69400
2009
206244
22916
229161
2010
50665
5629
56295
2011
55741
6193
61935
6
2012
51678
5741
57420
2013
64951
7216
72168
8
2014
79498
8833
88332
2015
193979
21553
215533
10
2016
120682
13409
134092
11
2017
111024
12336
123361
12
2018
123891
13765
137657
13
2019
142767
15862
158630
14
2020
166014
18445
184460
15
2021
201241
22360
223602
16
2022
89765
9973
99739
total
2007-2022
1852150
205786
2057952
20


Reviews:
[{"review_id": "mur1fn2q4G", "reviewer": "Reviewer_Nwuj", "summary": "This paper considers domain generalization over temporal-drift data where the model is trained online and required to generalize to the unseen future domain. The proposed method, Evolving Standardization (EvoS), assumes each domain follows a Gaussian distribution and utilizes transformers to capture the temporal drift by predicting the Gaussian mean and variance of the current domain given historical domains’ means and variances. Specifically, a multi-scale attention module (MSAM) is proposed, which utilizes a sliding window to aggregate domain statistics within a given time scope and makes the transformer predict based on the aggregated domains. The Gaussian statistics generator (namely, feature encoder) is shared over domains and trained in an adversarial way to achieve domain-invariant property. The proposed method is tested on several real-world temporal DG datasets and achieved good performance.", "questions": "Please refer to my “Weaknesses” section for the questions.\n\nMy final score will largely depend on the rebuttal and discussion with other reviewers. I am willing to increase my score if my questions are adequately addressed.", "limitations": "As far as I have checked, I did not find potential negative societal impacts in this work. I have listed my concerns in the \"Weaknesses\" section.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed problem, which seems like a combination of evolving/temporal domain generalization and online continual learning, is interesting and novel.\n\n2. The proposed method is shown to achieve optimal performance over various existing methods.\n\n3. The paper is in general clearly written and easy to follow.", "weaknesses": "1. The technical contribution is incremental. The single-scale attention is a review of transformer models and domain-invariant adversarial loss is a common tool in either DA/DG works. The key idea of the multi-scale attention module, which is a sliding window-based aggregation of multiple domains’ statistics, is relatively straightforward. Probably some theoretical analyses such as the generalization error bound of the proposed method on the future domain can strengthen the overall technical contribution.\n\n2. The usage of the memory pool is similar to episodic memory in continual learning replay-based methods, and I am wondering about the memory complexity/cost, especially considering the case where the number of temporal domains is large. Also, in online continua learning episodic memory is typically assumed fixed and small to guarantee practical interests, and I am curious if a similar assumption is made in this work.\n\n3. The baselines from the continual learning domain used in this paper are not SOTA (the latest one is A-GEM from 2019), and I encourage the author to compare them with more recent SOTA CL methods. In addition, continual domain adaptation is also a closely related area to this paper, and some methods such as CIDA [1] and CDOT [2] are also interesting to explore.\n\n4. I would encourage the author to consider visualizing the trained model in a gradual manner, such as the visualization of the decision boundary on the Rotated 2-Moons dataset in papers of e.g., CIDA, GI, and DRAIN. Such visualization can serve as qualitative analysis and better demonstrate if the method can truly capture the underlying temporal drift of data.\n\n[1] Wang, Hao, Hao He, and Dina Katabi. \"Continuously indexed domain adaptation.\" Proceedings of the 37th International Conference on Machine Learning. 2020.\n\n[2] Jimenez, G. O., Gheche, M. E., Simou, E., Maretic, H. P., and Frossard, P. Cdot: Continuous domain adaptation using optimal transport. arXiv preprint arXiv:1909.11448, 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your efforts in reviewing the paper. Below, we respond to your questions in detail.\n\n> **Q1:** The technical contribution is incremental.\n\n**A1:** Thanks. Firstly, although transformers can model sequential relationships, they require inputs themselves to be of sequentiality, like sentence words in NLP or tailored image patches in CV for ViT. In our case, sequentiality spans domains, not within a domain. Directly using transformers on the current domain data fails to capture evolving temporal patterns. Our efficient and novel solution is to model domain distribution using deep feature stats (mean, variance). We then use these statistics from seen domains as transformer inputs to generate next domain stats. Such way captures evolving patterns and meanwhile avoids heavy storage burden for historical data.\n\nSecondly, we further design the simple yet effective Multi-Scale Attention Module (MSAM) to accommodate and integrate the evolving pattern with different length of observation intervals, which has not been considered by previous temporal DG methods GI [2] and DRAIN [3]. Though the implementation of MSAM is relatively straightforward, the results of variant H *vs.* EvoS in the ablation study of the paper show that MSAM is indeed beneficial.\n\nThirdly, while domain-invariant adversarial loss is prevalent in DA/DG works, its application is challenging in our context due to inaccessible historical domains. We cleverly address this by modeling domain distribution at the deep feature level and utilizing truncated-sampled features as proxies of historical domains. Moreover, we further alleviate distribution shift by standardizing features via predicted evolving statistics. This approach leverages temporally evolutionary pattern, an aspect overlooked by prior DA/DG methods.\n\nLastly, beyond our method, our another contribution is the introduced problem of Continual Domain Generalization over Temporal Drift (CDGTD). This is a more challenging and practical variant of DG, considering the **dynamic nature of training domains**, the **efficiency of frequent model updates** with newly collected domain data, and the **storage burden** for historical data.\n\n> **Q2:** The memory complexity/cost and memory pool size.\n\n**A2:** Thanks for your comment. Please refer to the global reply **R2** for the detailed memory complexity/cost. As for the memory pool size, since the dataset used in our paper has a moderate number of domains and the memory cost is small, we do not restrict the size of the memory pool $\\mathcal{M}$ in our experiments. Yet, as you concerned, a fixed memory pool size is practical when considering a lifelong process, i.e., $T \\rightarrow \\infty$. Thus, we additionally conduct the experiment, where the memory pool $\\mathcal{M}$ is implemented as a FIFO queue with different fixed size $L$. That is, only the statistics for up to the $L$ most recent historical domains can be stored. The results on Yearbook dataset in **Figure 2(a)** of the PDF file demonstrate that our method generally performs well and a relatively large memory pool size is better for temporal domain generalization. So in practice, we recommend that the memory pool size be as large as possible under affordable memory cost. In the revision, we will clarify this.\n\n> **Q3:** Results for more recent SOTA CL method and continual DA methods CIDA and CDOT.\n\n**A3:** Thanks for your advice. We have added the results of CIDA, CDOT and a more advanced CL method SGP (AAAI'23) [4] on dataset Yearbook and RMNIST in **Table 2** of the PDF file. From the results, we can observe that the three methods obtain inferior performance on future domains.\n\nFor CIDA, it simultaneously takes the sample and time (i.e., [x, t]) as the input and aims to learn time-invariant feature representations via a two-party adversarial game. In this method, the temporal distribution shift is assumed to be characterized by the value of time, discarding the other features's dependency on time and dependency on other confounding factors. As a result, it fails to accurately capture the complex temporal drift.\n\nFor CDOT, it applies the regularized optimal transport to transport the most recent labeled samples to the future using a learned coupling from past data. Its performance depends on how closely the transported images resemble the true target image, showing limited ability in addressing the persistent temporal drift.\n\nAnd for SGP [4], it combines orthogonal gradient projections with scaled gradient steps along significant gradient spaces from prior tasks to enhance new learning and minimize forgetting. Despite the same mode of sequential training, our goal differs from SGP.  Our EvoS focuses on generalizing to upcoming unseen domains, whereas SGP aims to excel in both past and present tasks. Consequently, lacking specific temporal drift handling, SGP exhibits lower performance on future domains.\n\nBy contrast, our EvoS shows superior performance, owing to the better modeling of the evolutionary pattern in temporal domains via MSAM and the mitigation of the distribution shift by standardizing features with generated statistics.\n\n> **Q4:** Visualization of the decision boundary on Rotated 2-Moons dataset.\n\n**A4:** Good suggestion. **Figure 1** of the PDF file gives the result. The 2-Moons dataset in GI [2] is used, where each moon consists of 100 instances, and 10 domains (0 to 9) are obtained by sampling 200 data points from the 2-Moons distribution and rotating them counter-clockwise in units of $18^\\circ$. In Figure 1, the model is sequentially trained until the $t$-th domain is finished, and then we visualize the decision boundary on current domain $\\mathcal{D}^t$ and the next future domain $\\mathcal{D}^{t+1}$, $t=5,6,7,8$. According to the result, the decision boundary generated by our method successfully adapts to future domains, showing that our method can capture the underlying temporal drift of data. In the revision, we will add this experiment."}, {"review_id": "FqbfOxAkQp", "reviewer": "Reviewer_P1pd", "summary": "This paper introduces the problem of continual domain Generalization over Temporal Drift (CDGTD), where the domain distribution gradually changes over time and the model needs to generalize to new domains in the near future with training domains sequentially arriving. And this paper also proposes an Evolving Standardization (EvoS) approach to predict and adapt the data distribution of future domains by utilizing a multi-scale attention module to generate the mean and standard deviation of features in current domain and conducting feature standardization based on generated statistics to mitigate the domain shift. Extensive experiments on both image recognition and text classification are done to validate the effectiveness of the proposed method.", "questions": "Minor issue, typos:\nLine 299: Finally, the results of variant H and EvoS compares ...\nLine 309 and 319: MASM -> MSAM\nLine 321: \\beta and \\lambda controls ...\n", "limitations": "Limitations of the proposed approach have been discussed in the appendix.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of the problem of continual domain generalization over temporal drift (CDGTD) is innovative and meaningful. Unlike existing temporal domain generalization approaches, CDGTD incorporates incremental scenarios that mimic real-world situations where training domains arrive sequentially. This assumption adds an extra challenge to generalization, as the model needs to mitigate catastrophic forgetting during the continual learning process.\n\n2. The proposed method EvoS is novel to me. Especially, the inclusion of multi-scale attention module to generate evolved feature statistics is intriguing, where sliding time windows of different lengths are cleverly used to integrate multiscale information for better characterization of evolving patterns.\n\n3. Overall, the quality of the paper is commendable. The authors have conducted a thorough ablation study to examine the impact of various components, and they have also made the code and datasets available in the supplementary material.\n\n4. The paper is mostly well-organized and well-written.\n\n5. The experimental results on image recognition and text classification datasets in the CDGTD setting demonstrate state-of-the-art generalization performance across a diverse range of tasks.\n\n", "weaknesses": "1. I am wondering whether the number of heads (n_h) and the feature dimension of heads (d_h) are the same in each attention module A_w, since the authors just simply describe the value of n_h and d_h in MSAM. Yet, there are multiple attention modules in MSAM and the input dimension of each attention module is different. More details are necessary to make it clearer.\n\n2. In Eq. (12) and (13), it seems that the random sampling process is only performed once at the beginning of each training phase. Wouldn't this reduce the diversity and representativeness of the proxy samples for historical domains?\n\n3. As the number of historical domains increases, will the adversarial training suffer from the imbalance problem? What if we randomly select one historical domain in each iteration to participate in the adversarial training?\n\n4. The specific training way of “Offline” and “IncFinetune” in Table 1 and 2 should be provided.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your efforts in reviewing the paper as well as your constructive comments. Below, we do our utmost to address your concerns.\n\n> **Q1:** I am wondering whether the number of heads ($n\\_h$) and the feature dimension of heads ($d\\_h$) are the same in each attention module $\\mathcal{A}\\_w$, since the authors just simply describe the value of $n\\_h$ and $d\\_h$ in MSAM. Yet, there are multiple attention modules in MSAM and the input dimension of each attention module is different. More details are necessary to make it clearer.\n\n**A1:** Thanks for pointing out this. Actually, in practical implementation, we use the same feature dimension of heads $d\\_h$ for each attention module $\\mathcal{A}\\_w$ in MSAM, $w=1,2,\\cdots, W$, while the number of heads in attention module $\\mathcal{A}\\_w$ is set to $w\\cdot n\\_h$ to accommodate the different input dimension of each attention module. For the specific values of $d\\_h$ and $n\\_h$, we have provided in Section D.2 of the appendix ($d\\_h=8$ for all datasets and $n\\_h=16$ for Yearbook, $n\\_h=32$ for RMNIST, $n\\_h=64$ for fMoW, $n\\_h=128$ for Huffpost and Arxiv). In the revision, we will make this implementation details more clear.\n\n> **Q2:** In Eq. (12) and (13), it seems that the random sampling process is only performed once at the beginning of each training phase. Wouldn't this reduce the diversity and representativeness of the proxy samples for historical domains?\n\n**A2:** Thanks. In fact, the random sampling process is performed in each iteration to provide diverse and plenty features for representing historical domains. Concretely, in each iteration, we will randomly sample a batch of features from every preserved domain distribution in the memory pool $\\mathcal{M}$, and use them for the adversarial training to mitigate the overfitting of the model to current domain and enhance the generalizability of the feature encoder. This point will be clarified in the revision.\n\n> **Q3:** As the number of historical domains increases, will the adversarial training suffer from the imbalance problem? What if we randomly select one historical domain in each iteration to participate in the adversarial training?\n\n**A3:** Thanks for your comment. Although the total number of sampled features is greater than the number of features from current domain, the averaging operation in computing the adversarial loss balances the contribution of each feature to the adversarial training, which avoids suffering from the quantity imbalance problem. As for the results when randomly selecting one historical domain distribution in each iteration to participate in the adversarial training, we provide them in **Table 1** of the uploaded PDF file, where one distribution is randomly selected from the memory pool $\\mathcal{M}$ in each iteration to sample a batch of $B$ features for calculating the adversarial loss $\\mathcal{L}_{adv}$. From the results, we see that randomly selecting one historical domain distribution performs worse. This may be because the feature space to be aligned frequently changes if using this manner, making the optimization challenging. By contrast, it is more stable to simultaneously leverage all the preserved domain distributions in the memory pool $\\mathcal{M}$ in each iteration for the adversarial training.\n\n> **Q4:** The specific training way of “Offline” and “IncFinetune” in Table 1 and 2 should be provided.\n\n**A4:** Thanks for your advice. \"Offline\" denotes merging all training domains into one domain and training the model on the merged domain with the cross-entropy loss for classification tasks. And \"IncFinetune\" represents incrementally fine-tuning the model, i.e., the model is sequentially trained on each training domain. In other words, at each timestamp, the model is initialized into the state at the previous timestamp and then fine-tuned using the domain at current timestamp. We will add a detailed description of the training for the two baselines in the revised paper.\n\n> **Q5:** Minor issue, typos: Line 299: Finally, the results of variant H and EvoS compares ... Line 309 and 319: MASM -> MSAM Line 321: \\beta and \\lambda controls ...\n\n**A5:** Thanks. We have proofread the paper carefully and revised the paper thoroughly to correct the typos."}, {"review_id": "ahMUYFxXCg", "reviewer": "Reviewer_uaS8", "summary": "This paper introduces a problem formulation for Continual Domain Generalization over Temporal Drift (CDGTD) and proposes the Evolving Standardization (EvoS) method to address the challenge of gradually shifting data distributions over time, aiming to generalize to unseen domains that are not too far into the future. The EvoS method characterizes the evolving pattern of feature distribution and mitigates distribution shift by standardizing features with generated statistics of the corresponding domain. It utilizes a multi-scale attention module (MSAM) to learn the evolving pattern under sliding time windows of varying lengths. MSAM can also generate statistics of the current domain based on the previous domains' statistics and the learned evolving pattern. The paper demonstrates the efficacy of EvoS through experiments on multiple real-world datasets, including images and texts.", "questions": "Please refer to the weakness section.", "limitations": "Please refer to the weakness section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper aims to addresses an important problem in machine learning, namely, Continual Domain Generalization over Temporal Drift. The proposed algorithm, EvoS, is a new approach which characterizes the evolving pattern of feature distribution and mitigates the distribution shift by standardizing features with generated statistics of the corresponding domain.\n2. The paper highlights the necessity of learning evolving patterns to address temporal distribution shift.\n3. The paper conducts experiments on multiple real-world datasets, including images and texts, to validate the efficacy of the proposed EvoS method.", "weaknesses": "1. The paper presents a setting where domains arrive sequentially, and models can only access the data of the current domain. However, the authors argue that we should learn from the current source domain to adjust models and generate target domains. This setting contradicts the idea of Domain Generalization, where data from multiple domains may be available simultaneously, or models can access historical data from previous domains.\n\n2. However, the authors of the paper do not provide sufficient context or motivation for the research problem and fail to distinguish this setting from continuous learning and test-time adaptation.\n\n3. The EvoS method introduced in this paper is notably more complex than other baselines. However, the authors do not provide a detailed analysis of the computational requirements of the proposed method.\n\n4. EvoS outperforms other baselines in its tailor-made tasks, which make the efficacy of EvoS unconvincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your efforts in reviewing the paper. Below, we address your concerns in detail.\n\n> **Q1:** This setting contradicts the idea of Domain Generalization.\n\n**A1:** Thanks. Firstly, we want to clarify that our CDGTD is a challenging and practical variant of conventional DG. Hence, its setting naturally differs.\n\nPast DG methods focus on static, discrete scenarios with fixed domains and sudden shifts among them. They train a model on multiple domains simultaneously offline, striving for ideal but hardly achievable generalization on any unsee domain. In real world, new data continuously arises over time. A realistic way is to leverage fresh domain along with old domains to enhance model generalization. One way is to store past domains and re-train the model using standard DG methods. Yet, this is inefficient and doesn't allow reuse of previous model. Besides, retaining all historical domains requires substantial memory, especially in scenarios with rapid data accumulation.\n\nConsidering the limitations, we draw inspiration from continual learning (CL) to consider DG in a dynamic and continuous configuration. In CDGTD, the training domain evolves dynamically over time following some patterns, and domains are assumed to arrive sequentially to mimic the realistic scenario where new training domain emerges, while the access to only current domain minimizes storage cost. Like CL, the model starts from the previous timestamp's state and incrementally trains with current domain. This makes previously trained model reusable and boosts training efficiency.\n\nOverall, CDGTD aims at a more practical DG scenario that simultaneously considers the **dynamics of training domains**, **training efficiency** for frequently updating model for better generalization and **storage burden**.\n\n> **Q2:** Motivation for CDGTD and differences with continuous learning (CL) and test-time adaptation (TTA).\n\n**A2:** Thanks. Actually, in the introduction, we have explained why CDGTD is introduced. For your convenience, we summarize it below.\n\n**Motivation of CDGTD.** In standard DG, models are trained offline on fixed source domains to achieve broad generalization on any unseen domain. However, such ideal generalization is tough. In real world, new data continually emerges over time, offering a chance to enhance the generalization of previously trained model. But standard DG methods struggle to update models with fresh data efficiently. Their static domain setup and offline training mode rquire starting model training anew with new and prior data, leading to low efficiency and high storage cost for historical data. Instead, a more practical way is to employ the training paradigm of CL, but with the goal to generalize on future unseen domains. This way reduces storage cost and enhances training efficiency. Thus, we introduce CDGTD to efficiently and effectively address temporal DG.\n\n**Differences with CL.** The biggest difference is the objective. CL aims to learn new tasks while retaining performance on old tasks, whereas CDGTD prioritizes generalization on future unseen domains. These different goals yield distinct challenges. CL faces catastrophic forgetting of task-specific knowledge, while CDGTD tackles modeling underlying evolutionary patterns of temporal domains, and how to utilize these patterns to mitigate distribution shifts in forthcoming domains.\n\n**Differences with TTA.**  One difference is the focused optimization phase. TTA optimizes the source-pretrained model during testing phase with test data, while CDGTD optimizes the model during training phase with sequential training domains. Another difference is the data. TTA's arriving test data usually comes from the same domain or distribution, while CDGTD's test domain evolves over time, showing distribution shifts. Moreover, TTA emphasizes in-time adaptation using test data, while CDGTD focuses on generalization without using test data.\n\n> **Q3:** Analysis of the computational requirements of the method.\n\n**A3:** Thanks. Please refer to the global reply **R2** for the detailed analysis of memory and time complexity.\n\n> **Q4:** EvoS outperforms other baselines in its tailor-made tasks, which make the efficacy of EvoS unconvincing.\n\n**A4:** Firstly, the challenging CDGTD remains mostly unexplored. It is inevitable to run related methods in our experimental setup for comparisons. To ensure fairness, we have applied the same training and evaluation settings for compared methods.\n\nSecondly, as described in Section 4.1, our experimental tasks mainly comes from the existing Wild-Time benchmark (NeurIPS'22) [1]. This benchmark offers multiple **real-world** datasets displaying distribution shifts over time, including the Yearbook, fMoW, Huffpost, and Arxiv datasets we use. By contrast, prior temporal DG methods GI [2] and DRAIN [3] use either synthetic (e.g., 2-Moons) or small (e.g., Shuttle) datasets. Thus, we opt to assess performance on more challenging and realistic datasets.\n\nThirdly, the evaluation strategy is not specially tailor-made by us. It is from the Eval-Fix strategy in Wild-Time [1]. Specifically, given a domain sequence of length $\\mathcal T$ and a split timestamp $T\\_s$, the model trains on $\\mathcal{D}^1$ to $\\mathcal{D}^{T_s}$ sequentially and evaluates on future domains $\\mathcal{D}^{T\\_s+1}$ to $\\mathcal{D}^{\\mathcal T}$. Under Eval-Fix, average performance $OOD_{avg}$ and worst performance $OOD_{worst}$ are defined as\n\n$OOD_{avg} = \\frac{1}{\\mathcal T - T\\_s} \\sum_{i=T\\_s+1}^{\\mathcal T} {Acc(\\mathcal{D}^i)}, OOD_{Worst} = \\min_{i\\in \\\\{T_s+1, \\cdots, \\mathcal T\\\\}} {Acc(\\mathcal{D}^i)}.$\n\nFinally, to address your concerns about convincingness, we present the generalization results on the last domain for 2-Moons, ONP, and Elec2 datasets from DRAIN [3] in **Table 4** of the PDF file. Notably, all baselines' results are reported from DRAIN. The outcomes show EvoS's superior performance on these non-tailored tasks, affirming its effectiveness."}, {"review_id": "IthFpFa1SL", "reviewer": "Reviewer_YTjU", "summary": "Mitigating the temporal distribution shift problem is meaningful in realistic. This paper introduces the problem of continual domain generalization over temporal drift (CDGTD), aiming to address the issue of potential temporal distribution shifts in unseen test domains. The authors propose an Evolving Standardization method that utilizes a multi-scale attention module to forecast the distribution of the unseen test domain. They then mitigate the distribution shift by standardizing features using the generated statistics of the test domain. To the best of my knowledge, the method is novel and shows promising results in the experiments section. ", "questions": "1. Figure 3 (b) shows that the method is not sensitive to hyperparameters \\alpha and \\lambda on yearbook datasets which is pretty strange to me. Do the authors have some intuitions about that? If I am a user using your method, Could the authors provide suggestions or considerations for selecting appropriate values for these hyperparameters? In particular, I am curious about the rationale behind the specific choices of {0.1, 0.5, 1.0, 1.5, 2.0} for the ablation experiment.\n\n2. What is the difference between your experimental settings and wilds-time's eval-stream? Why don't directly follow wilds-time for the experiments?\n", "limitations": "NIL", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper exhibits a commendable structure and offers clear explanations throughout.\n2. The authors conduct ablation studies to examine the individual components of their proposed method.\n3. To the best of my knowledge, the method presented in this paper is novel.\n4. Experiments results confirm the proposed method works well.", "weaknesses": "1. The authors only selected a subset of datasets from the Wilds-Time benchmark for their experiments, it would be valuable to include results on MIMIC-Readmission and MIMIC-Mortality datasets as well, considering wilds-time is the most relevant benchmark for this problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Sincerely thanks for your efforts in reviewing the paper. Below, we respond to your questions in detail.\n\n> **Q1:** The used datasets are only a subset of datasets from the Wilds-Time benchmark. It would be valuable to include MIMIC-Readmission and MIMIC-Mortality datasets.\n\n**A1:** Thanks for your suggestion. Actually, we almost use all the provided datasets in the Wilds-Time benchmark, except for the MIMIC-Readmission and MIMIC-Mortality datasets. This omission is not deliberate. The reason is that they are restricted-access resources and we have some troubles in acquiring the credentialed access to the two medical datasets. Since Yearbook, fMoW, Huffpost and Arxiv are all from the Wilds-Time benchmark and our method has achieved comparable and even superior performance on the four datasets, we think these results are sufficient to validate the effectiveness of our method. Yet, of course, there is no doubt that more results are more convincing, and we will leave the successful application for the credentialed access and the results for the two medical datasets as future work.\n\nAs a compensation for the missing results of these two datasets, we additionally run our method on some other datasets (2-moons, Online News Popularity (ONP) and Electrical Demand (Elec2)) that are used by previous temporal DG methods GI [2] and DRAIN [3]. These datasets are also of distribution shifts over time. Please refer to the global reply **R1** for the detailed dataset description. **Table 4** in the PDF file gives the results, where the misclassification errors of baselines are reported from DRAIN [3]. We see that our method EvoS still outperforms the most recent method DRAIN.\n\n> **Q2:** Explanation for the insignificant hyperparameter sensitivity and suggestion for hyperparameter values.\n\n**A2:** Thanks. The hyperparameters $\\alpha$ and $\\lambda$ control sampling truncation range and the tradeoff of adversarial loss $\\mathcal{L}\\_{adv}$, respectively. For the normal distribution, values within one standard deviation from the mean make up 68.27%, two deviations account for 95.45%, and three deviations cover 99.73%. Since $\\alpha=3$ is akin to no truncation (Variant G in the ablation study of the paper), we limit $\\alpha$ to 2 in the sensitivity experiment, using an interval of 0.5 for other values. For $\\lambda$, as $\\mathcal{L}\\_{adv}$ is of similar magnitude as cross-entropy loss $\\mathcal{L}\\_{ce}$, we vary $\\lambda$ by 0.5 intervals in both directions around $\\lambda=1.0$.\n\nActually, in the original Figure 3(b) of the paper, sensitive regions exist (e.g., $\\lambda \\in [1.0, 2.0]$). However, due to the narrow value range, sensitivity might not be evident. In **Figure 2(b)** of the PDF file, we extend hyperparameter ranges to $\\alpha \\in \\\\{0.1, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0\\\\}$ and $\\lambda \\in \\\\{0.1, 0.5, 1.0, 1.5, 2.0, 5.0, 10.0, 15.0, 20.0\\\\}$. These new results exhibit similar conclusions that EvoS is more sensitive to $\\lambda$ than $\\alpha$, and large $\\lambda$ values significantly worsen performance. Thus, in practice, we recommend selecting $\\lambda$ from a range $\\leq$1 through validation. For $\\alpha$, its effect is modest; users can directly use $\\alpha=1$, which balances diversity and representativeness, as about 68.27% of the data lie within one standard deviation from the mean in a normal distribution.\n\n> **Q3:** Difference between our experimental setting and wilds-time's eval-stream.\n\n**A3:** Thanks. Wild-Time [1] provides two evaluation strategies: Eval-Fix and Eval-Stream, and our experimental setting uses the Eval-Fix. Eval-Fix denotes that the model is evaluated on a single, fixed train-test time split. This evaluation strategy offers a quick evaluation protocol and is the primary evaluation strategy in Wild-Time. Thus, we adopt the Eval-Fix strategy in our experiments.\nSpecifically, given a sequence of domains with total length $\\mathcal{T}$ and the split timestamp $T\\_s$, the average performance $Avg\\_{fix}$ (i.e., \"OOD avg.\" in our paper) and worst performance $Worst\\_{fix}$ (i.e., \"OOD worst\" in our paper) under the strategy of Eval-Fix are defined as\n\n$Avg\\_{fix} = \\frac{1}{K} \\sum_{i=T\\_s+1}^{T\\_s+K} {Acc(\\mathcal{D}^i)}, Worst\\_{fix} = \\min\\_{i\\in \\\\{T\\_s+1, \\cdots, T\\_s+K\\\\}} {Acc(\\mathcal{D}^i)},$\n\nwhere $K=\\mathcal{T} - T\\_s$ is the number of future domains to be evaluated, and $Acc(\\mathcal{D}^i)$ is the accuracy on domain $\\mathcal{D}^i$.\n\nAs for Eval-Stream, it denotes the evaluation with data stream, i.e., the model is evaluated at each timestamp using the average and worst performance on the next $K$ timestamps. Concretely, given a sequence of domains with total length $\\mathcal{T}$, the average performance $Avg\\_{stream}$ and worst performance $Worst\\_{stream}$ under the strategy of Eval-Stream are defined as\n\n$Avg\\_{stream} = \\frac{1}{\\mathcal{T}-K} \\sum_{i=1}^{\\mathcal{T}-K} \\frac{1}{K} \\sum_{j=i+1}^{i+K} Acc_{i}(\\mathcal{D^j}), Worst\\_{stream} = \\frac{1}{\\mathcal{T}-K} \\sum_{i=1}^{\\mathcal{T}-K} \\min_{j\\in \\\\{i+1, \\cdots, i+K\\\\}} Acc_{i}(\\mathcal{D^j}),$\n\nwhere $Acc_{i}(\\mathcal{D^j})$ is the accuracy on domain $\\mathcal{D^j}$ when using the model at the $i$-th timestamp. Hence, the Eval-Fix that we adopt can be viewed as a single timestamp evaluation within Eval-Stream, where the model is only evaluated at $T\\_s$ and $K=\\mathcal{T}-T\\_s$.\n\nTo further verify the effectiveness of our method, in **Table 3** of the PDF file, we additionally provide the result when using the Eval-Stream evaluation strategy. Due to time limitations, we just provide results on Yearbook and Huffpost datasets. According to the result, we see that EvoS still outperforms other baselines, showing that our method is better at handing the problem of continual domain generalization over temporal drift."}]

Author rebuttal:
Sincerely thank all reviewers for the efforts in reviewing our paper and the constructive suggestions. We are more than encouraged that reviewers find 
* our proposed problem of Continual Domain Generalization over Temporal Drift (CDGTD) to be **interesting** and **novel** (*Reviewer Nwuj*), **innovative** and **meaningful** (*Reviewer P1pd*), **important** (*Reviewer uaS8*),
* our method to be **novel** and **intriguing** (*Reviewer P1pd*), **new** and with **efficacy** (*Reviewer uaS8*), **novel** and **promising** (*Reviewer YTjU*),
* our paper to be **clearly written** (*Reviewer Nwuj*) and with **commendable structure** (*Reviewer YTjU*) and **quality** (*Reviewer P1pd*),
* the ablation study to be **thorough** (*Reviewer P1pd*),
* and the performance to be **state-of-the-art** across a diverse range of tasks (*Reviewer P1pd*).

As for the concerns and suggestions of each reviewer, we have done our utmost to address them and responded to each of them in detail. And a one-page PDF file has been uploaded containing the relevant figures and tables mentioned in the responses. Please refer to the PDF file for detailed results, if needed. Below are some global replies (**R**) that may be used in the  separate response to different reviewers:

> **R1: Introduction to the dataset used in the rebuttal.**

Apart from the dataset in our paper, we additionally run our method on below datasets with temporal shifts that are used by previous temporal DG methods [2, 3]. The description of these datasets are given as below.
* **2-Moons** is a variant of the 2-entangled moons dataset by rotating data counter-clockwise in units of $18^\circ$ to construct 10 domains, where the rotation angle is used to simulate the temporal shift.
* **Online News Popularity (ONP)** summarizes a heterogeneous set of features about articles published by Mashable in a period of two years. The dataset is split into 6 domains by time and the goal is to predict the number of shares in social networks (popularity).
* **Electrical Demand (Elec2)** contains information about the demand of electricity in a particular province. Following [2, 3], the first 30 domains are used (two weeks as one time domain) and the task is to predict if the demand of electricity in each period (of 30 mins) was higher or lower than the average demand over the last day.

For these datasets, following [2, 3], we use the last domain for testing and the rest for training.

> **R2: The complexity anlysis of our method.**

**Memory complexity** mainly comes from the memory pool $\mathcal{M}$. Assuming that there are $T$ historical domains and the feature dimension is $d_f$, then the memory complexity of the memory pool is $\mathcal{O}(T\cdot d_f)$. In practice, the dimension of pooled features is usually much smaller than that of original inputs after processing by the deep neural network. For example, the dimension of an image in the fMoW dataset is $224\times 224 \times 3 = 150,528$, while the dimension of pooled features in DenseNet-121 is $1,024$, about $0.007$ times of the former. Besides, only two vectors need to be stored per domain. Hence, the memory cost of $\mathcal{M}$ is relatively small, compared with sample replay-based CL methods. Concretely, the memory cost of $\mathcal{M}$ for the relatively large dataset fMoW is $32$ MB, and the GPU memory consumption of the whole method EvoS increases by $0.35$ GB over IncFinetune (10.69 GB of IncFinetune and 11.04 GB of EvoS).

**Time complexity** mainly comes from the multi-scale attention module (MSAM). Taking one of the attention module $\mathcal{A}_w$ in MSAM as an example, we assume that $d_f$ is the feature dimension of input tokens, $d_h$ and $n_h$ are the feature dimension and the number of heads, and $n_i$ is the number of input tokens in $\mathcal{A}_w$. Then the time complexity comprises:

* the transformation of input tokens to their query, key and value embeddings: $\mathcal{O}(n_i \cdot d_f \cdot d_h \cdot n_h)$,
* the calculation of attention weight matrix: $\mathcal{O}(n_i \cdot d_h \cdot n_i \cdot n_h)$,
* the multiplication of attention weight matrix and value matrix: $\mathcal{O}(n_i \cdot n_i \cdot d_h \cdot n_h)$,
* convert the feature dimension of attended value embeddings into the input dimension: $\mathcal{O}(n_i \cdot d_h \cdot d_f \cdot n_h)$.

Thus, the time complexity is $\mathcal{O}(n_i \cdot d_f \cdot d_h \cdot n_h) + \mathcal{O}(n_i \cdot d_h \cdot n_i \cdot n_h) + \mathcal{O}(n_i \cdot n_i \cdot d_h \cdot n_h) + \mathcal{O}(n_i \cdot d_h \cdot d_f \cdot n_h)\approx\mathcal{O}((n_i^2 + n_i \cdot d_f) \cdot (\cdot d_h \cdot n_h)).$

Since $n_i$ will be no larger than the number of training domains $T$ and $d_h \cdot n_h$ usually sets to $d_f$ in transformers, the time complexity of MSAM can be roughly approximate as $\mathcal{O}(W \cdot (T^2 d_f + T \cdot d_f^2))$, where $W$ is the number of multi-head attention modules in MSAM. In the implementation, $W$ is set to a relatively small value ($W=3$) in our paper and the time complexity is acceptable.

**Global references:**

[1] Wild-time: A benchmark of in-the-wild distribution shift over time. In NeurIPS, 2022.

[2] Training for the future: A simple gradient interpolation loss to generalize along time. In NeurIPS, 2021.

[3] Temporal domain generalization with drift-aware dynamic neural networks. In ICLR, 2023.

[4] Gobinda Saha, Kaushik Roy. Continual Learning with Scaled Gradient Projection. AAAI, 2023.

Meta-review: The paper proposes a new practical setting for continual domain generalization where data distribution is evolving over time. The authors have proposed a novel method - multi scale attention. The method is efficient and shows better performance over existing baselines and methods.  The paper is well written and well presented. All reviewers recommend accepting the paper.
Decision: Accept (poster)