Title: Evolving Standardization for Continual Domain Generalization over Temporal Drift
Venue: NeurIPS 2023
Accepted: yes
Keywords: domain generalization

sequential learning

temporal drift

feature standardization
Abstract: The capability of generalizing to out-of-distribution data is crucial for the deployment of machine learning models in the real world. Existing domain generalization (DG) mainly embarks on offline and discrete scenarios, where multiple source domains are simultaneously accessible and the distribution shift among domains is abrupt and violent. Nevertheless, such setting may not be universally applicable to all real-world applications, as there are cases where the data distribution gradually changes over time due to various factors, e.g., the process of aging. Additionally, as the domain constantly evolves, new domains will continually emerge. Re-training and updating models with both new and previous domains using existing DG methods can be resource-intensive and inefficient. Therefore, in this paper, we present a problem formulation for Continual Domain Generalization over Temporal Drift (CDGTD). CDGTD addresses the challenge of gradually shifting data distributions over time, where domains arrive sequentially and models can only access the data of the current domain. The goal is to generalize to unseen domains that are not too far into the future. To this end, we propose an Evolving Standardization (EvoS) method, which characterizes the evolving pattern of feature distribution and mitigates the distribution shift by standardizing features with generated statistics of corresponding domain. Specifically, inspired by the powerful ability of transformers to model sequence relations, we design a multi-scale attention module (MSAM) to learn the evolving pattern under sliding time windows of different lengths. MSAM can generate statistics of current domain based on the statistics of previous domains and the learned evolving pattern. Experiments on multiple real-world datasets including images and texts validate the efficacy of our EvoS.

Evolving Standardization for Continual Domain Generalization over Temporal Drift

Abstract

The capability of generalizing to out-of-distribution data is crucial for the deployment of machine learning models in the real world. Existing domain generalization (DG) mainly embarks on offline and discrete scenarios, where multiple source domains are simultaneously accessible and the distribution shift among domains is abrupt and violent. Nevertheless, such setting may not be universally applicable to all real-world applications, as there are cases where the data distribution gradually changes over time due to various factors, e.g., the process of aging. Additionally, as the domain constantly evolves, new domains will continually emerge. Re-training and updating models with both new and previous domains using existing DG methods can be resource-intensive and inefficient. Therefore, in this paper, we present a problem formulation for Continual Domain Generalization over Temporal Drift (CDGTD). CDGTD addresses the challenge of gradually shifting data distributions over time, where domains arrive sequentially and models can only access the data of the current domain. The goal is to generalize to unseen domains that are not too far into the future. To this end, we propose an Evolving Standardization (EvoS) method, which characterizes the evolving pattern of feature distribution and mitigates the distribution shift by standardizing features with generated statistics of corresponding domain. Specifically, inspired by the powerful ability of transformers to model sequence relations, we design a multi-scale attention module (MSAM) to learn the evolving pattern under sliding time windows of different lengths. MSAM can generate statistics of current domain based on the statistics of previous domains and the learned evolving pattern. Experiments on multiple real-world datasets including images and texts validate the efficacy of our EvoS.

1 Introduction

In real-world applications, the assumption that training and testing data conform to the same distribution, a prerequisite for the success of contemporary deep learning methods, is seldom valid. A prime example of this can be found in autonomous driving, where a vehicle may traverse environments that switch from daylight to nightfall or from urban to rural. As environmental conditions change, the issue of distribution shift [5, 4, 51, 62] arises. Moreover, directly utilizing a model trained on in-distribution (ID) data in an out-of-distribution (OOD) context often results in catastrophic performance deterioration. Consequently, ensuring that models perform well on OOD data has emerged as a crucial challenge for the widespread deployment of machine learning models in the real world.

To cope with the distribution shift, two dominant paradigms have been systematically explored depending on the availability of target (test) domain. One is domain adaptation (DA), which aims to assist the model learning on an unlabeled or label-scarce target domain by transferring the knowledge

from a related and label-rich source domain [15, 34, 29, 35]. Yet, target data are not always known or available in advance. The other is domain generalization (DG), the goal of which is to learn a model capable of generalizing to any unseen domain by using date from multiple related but distinct source domains [62, 36, 27, 63, 26]. Though the second paradigm DG has attained some encouraging outcomes, its configuration is limited to offline and discrete scenarios where multiple source domains can be accessed simultaneously, and the distribution shift among domains is sudden and severe. For instance, the prevalent benchmark PACS [25] in current DG methods comprises four distinct domain styles - 'Art', 'Cartoon', 'Photo' and 'Sketch'. Nevertheless, this type of configuration may not be suitable for all real-world applications. There are also some cases that the data distribution gradually evolves over time, the distribution shift arising from which is referred to as temporal drift [56].

On one hand, the real-world scenarios often exhibit underlying evolutionary patterns [59, 44]. For example, annual or monthly weather data can be utilized for weather forecasting [12]. The evolutionary patterns can be exploited to enhance generalization capabilities towards future unseen domains that are not too distant. However, current DG methods often fail to consider these patterns, leading to suboptimal performance. On the other hand, since data distribution is constantly evolving, new domains will continue to emerge. Consequently, it is imperative to efficiently utilize these new domains to further enhance the model's performance for practical applications. For instance, in the context of advertisement recommendation, user browsing data for various products continually surfaces. How can we leverage these newly collected data to enable more accurate advertisement recommendations tailored to each user's preferences in the days to come? One simple approach is to store data from previous domains and retrain the model using both the new and previous domains with the existing DG techniques. However, such way may consume huge training resources due to the accumulation of data and is of low efficiency, especially for scenarios with rapid data accumulation.

In this paper, we formulate the aforementioned problems as Continual Domain Generalization over Temporal Drifts (CDGTD), where the data distribution gradually drifts with the passage of time and the domain arrives sequentially. And the goal of CDGTD is to generalize well on future unseen domains that are not too far under the circumstance that only current domain is available at current times point while data from previous domains are inaccessible. Although some temporal / evolving DGmethods [44, 59, 3, 41] have been proposed to handle the temporal drift, most of them [59, 44, 41] works in non-incremental setting, i.e., multiple domains can be accessed simultaneously. Instead, we design an Evolving Standardization (EvoS) approach specialized for CDGTD.

For CDGTD, there are two main challenges: how to characterize the evolving pattern in the incremental setting and how to achieve generalization using the learned evolving pattern. For the former, we draw inspiration from the sequence modeling capability of transformers [31] and design a multi-scale attention module (MSAM) to learn the evolving pattern underlying the feature distribution of domains. Specifically, we store the statistics (i.e., mean and variance of features) of previous domains and use sliding time windows of different lengths over the statistic tokens to obtain multi-scale information. Then multi-scale statistic tokens are fed into MSAM to generate statistics of current domain. MSAM is trained over the whole sequence of domains to learn the evolving pattern. Here, we integrate multiscale information, with the consideration that some evolving patterns may be better characterized at different time intervals, e.g., seasonal climate and daily weather. For the latter challenge, in order to mitigate the temporal drift, each domain is transformed into a common normal distribution by conducting feature standardization with the generated statistics of corresponding domain. Besides, considering that the feature encoder may suffer from catastrophic forgetting and overfitting to current domain, we constrain it to learn a shared feature space among domains via the adversarial learning.

Contributions: 1) We formulate a promising but challenging problem of continual domain generalization over temporal drift (CDGTD), which has seldom been explored, compared with traditional DG. 2) An evolving standardization (EvoS) approach is specially proposed for CDGTD, which can characterize the evolving pattern and further achieve generalization by conducting the feature standardization. 3) Experiments on multiple real-world datasets with different models verify the effectiveness of EvoS and the flexibility to be applied on different models.

2 Related Work

Domain Generalization (DG) aims to learn a model that can generalize well to any unseen target domains by leveraging data from multiple source domains. In recent years, a wide range of DG

methods have been proposed [6, 39, 27, 26, 63, 19, 37]. According to the strategies of improving generalization, existing DG methods roughly fall into the three categories. Representation learning based methods [7, 39, 50, 27, 27, 2, 17, 42] aim to learn domain-invariant or domain-shared representations to enable models to generalize to unseen target domains. Data augmentation/generation based methods [53, 61, 63, 57, 32] focus on manipulating inputs to facilitate learning general representations. Differently, other DG methods instead employ general learning strategies like meta-learning [26] and ensemble learning [21] to improve generalization ability. With full access to all source domains, these DG methods have achieved promising performance on unseen domains with abrupt and violent distribution shifts. Unfortunately, without the consideration of the evolutionary pattern of domains over time, existing DG methods are usually less efficient under the scenario of temporal drift.

Continual Learning (CL) is a machine learning paradigm, where models learn continuously from a stream of tasks over time and meanwhile try to retain performances on all seen tasks [22, 1, 24, 9, 46, 47]. Existing CL methods can be roughly categorized into replay-based [46, 10, 52], regularization-based [22, 60, 30] and structure-based [45, 14] methods. In addition, the combination of CL and DA has attracted lots of attention [23, 17, 33, 43, 55]. Yet, combining CL and DG remains underdeveloped. In this work, we propose CDGTD which continually trains the model on sequential domains, but the goal is to generalize well on novel domains in the near future. This distinct objective from CL yields different challenges: the modeling of underlying evolutionary patterns of temporal domains and how to utilize these patterns to mitigate the distribution shifts in forthcoming domains.

Test Time Adaptation (TTA) focuses on adapting the source-pretrained model during testing phase with test data [40, 49, 20, 54, 58]. For example, [49] corrects the activation statistics of batch normalization using test data. T3A [20] uses a back-propagation-free manner to adjust only the weights of the linear classifier during test time. By contrast, CDGTD optimizes the model during training phase with sequential domains and emphasizes on generalization without using test data.

Evolving / Temporal Domain Generalization has emerged in recent years, aiming to tackle the problem of generalizing on temporally drifted domains, where the environment evolves dynamically over time [41, 3, 44, 59]. GI [41] proposes a time-sensitive model architecture to capture the time-varying data distribution and the model is supervised with the first-order Taylor expansion of the learned function to advance the generalization in the near feature. DRAIN [3] launches a Bayesian framework to model the concept drift and utilizes a recurrent neural network to dynamically generate network parameters to adapt the evolving pattern of domains. LSSAE [44] incorporates variational inference to explore the evolving patterns of covariate shift and label shift in the latent space. The goal of learning evolutionary patterns from source domains and generalizing to domains in the near future is similar to our EvoS. The main difference is that the model in our EvoS is incrementally trained on sequentially arriving domains, considering the low efficiency of 'offline' training with accumulated domains. By contrast, the aforementioned methods [41, 44] require multiple source domains to be simultaneously available. Besides, the Taylor expansion in [41], variational inference in [44] and network parameters generation in [3] make them hard to expand on large models.

3 Evolving Standardization for Continual Domain Generalization over Temporal Drift

3.1 Problem Formulation of CDGTD

Here, we take the C -class classification problem as an example, where X and Y denote the data and label space, respectively. Suppose that T source (training) domains {D 1 , D 2 , · · · , D T } sequentially arrive, which are sampled from distributions at T different times points t 1 < t 2 < · · · < t T . At time point t t , t ∈ { 1 , 2 , · · · , T } , only the domain D t = { x t i , y t i } N t i =1 is accessible, where x t i ∈ X , y t i ∈ Y and N t is the number of training samples in D t . Previous and future domains are unavailable. In addition, as previous temporal/evolving DG methods [44, 41, 3], we further assume that the data distribution P ( X,Y ) of domains evolves temporally, i.e., the distribution of domains changes along the time following certain patterns. The goal of CDGTD is to enable the model, composed of a feature encoder E : x → f ∈ R d f ( d f is the dimension of features) and a classifier C : f → y ∈ { 0 , 1 , · · · , C -1 } , to generalize well on K unseen target (test) domains in the near future: {D T + k } K k =1 . To this end, two main challenges need to be addressed. One is to characterize the evolving pattern of domains, which we address by designing a multi-scale attention module (MSAM)

feature

encoder

ℰ

classifier

𝒞

discriminator

𝔇

𝒟

!

Feature Standardization

𝒛

!

"

=

𝒇

!

"

- ( 𝝁

"

( 𝝈

"

=

𝒇

!

"

- ( 𝝁

"

exp(( 𝒂

"

)

𝒇

!

"

𝒛

!

"

ℒ

#$

"

ℒ

%&'

"

𝒙

!

"

…

…

…

…

𝒟

!"#

𝒟

!"$

…

𝑡

…

𝒔

(

=

𝒂

(

𝝁

(

,

𝒔

)

𝒔

*

…

𝒔

*+(

𝒔

",*,(

𝒔

",*

𝒔

",)

𝒔

(

…

𝒔

",(

…

concat

concat

concat

concat

attention

module

𝒜

(

attention

module

𝒜

*

attention

module

𝒜

-

…

average operation:

6 𝒔

"

=

(

-

∑

*.(

-

6 𝒔

*

"

6 𝒔

(

"

6 𝒔

*

"

6 𝒔

" -

6 𝒔

"

…

𝒔

)

=

𝒔

",(

=

𝒂

)

𝝁

)

,

𝒂

",(

𝝁

",(

,

ℒ

/"%0

"

ℒ

#10

", *

…

𝒔

",*+(

=

𝒂

",*+(

𝝁

",*+(

,

memory pool

𝓜

…

…

multi-head self-attention module

𝒜

*

…

average operation:

9 𝒔

*

",*+(

=

(

",*

∑

!.(

",*

9 𝒔

*

!

*

(

*

)

*

",*,(

*

",*

…

9 𝒔

*

(

9 𝒔

*

)

9 𝒔

*

",*,(

9 𝒔

*

",*

…

…

9 𝒔

*

",*+(

6 𝒔

*

",*+(

6 𝒔

*

",(

6 𝒔

*

"

…

split

stop gradient of

ℒ

!"

#

stop gradient of

ℒ

$#%&

#

{𝒇

'

(

(

, 𝒇

'

)

(

, ⋯ , 𝒇

'

*

(

}

{𝒇

'

(

)

, 𝒇

'

)

)

, ⋯ , 𝒇

'

*

)

}

{𝒇

'

(

#+(

, 𝒇

'

)

#+(

, ⋯ , 𝒇

'

*

#+(

}

…

…

Multi-scale Attention Module

(MSAM)

statistic token

concatenated-statistic token

sliding time window

inaccessible

historical domain

statistic token

( 𝝁

"

( 𝒂

"

sample from

historical feature

distributions

Generate statistic token with attention module

𝓐

𝝎

inspired from the sequence modeling ability of transformers. The other is the generalization, which is realized by transforming domains into a common normal distribution via the feature standardization.

3.2 EvoS: Evolving Standardization

Single-scale Attention. Inspired by the capability of the transformer to model the relationships among sequences [31], we introduce its attention mechanism to model the evolving pattern among sequential domains. Before presenting our multi-scale attention module (MSAM), we first introduce how single-scale attention works. Without loss of much generality, we assume that the feature distribution of domain D t follows the normal distribution, which is characterized by the mean vector µ t ∈ R d f and the standard deviation vector σ t ∈ R d f . Now, our goal turns out to be learning the evolving pattern of the feature statistics (i.e., µ t and σ t ). In practice, to ensure that σ t is non-negative, we choose to learn a t (the logarithm of σ t ). In this way, σ t = exp( a t ) is always non-negative. Besides, considering the unavailability of previous domains, we store the learned statistics at time point t t into a memory pool M for future uses once the training procedure finishes on domain D t . Having the feature statistics of previous t -1 domains ready, we then utilize the multi-head self-attention module A to generate the statistics of domain D t at time point t t .

Concretely, given statistic tokens { s i } t -1 i =1 of previous t -1 domains, where s i = [ µ i , a i ] ∈ R 2 d f is the token of concatenated statistics from domain D i , the output of A is expressed as

where ˆ s i is the i -th output statistic token of A , n h and d h are the number and feature dimension of heads in the multi-head self-attention module and W fc is the learnable parameters of A to convert feature dimensions. SA i ( · ) is the self-attention of the i -th head, which operates as follows:

where W i qkv is the learnable parameters of the i -th head and S t -1 i,q , S t -1 i,k , S t -1 i,v are the query, key and value embeddings of S t -1 . Finally, we use the average of all output statistics tokens of the attention module A as the generated statistic token ˆ s t for time points t t :

By continuously generating new statistics from historical ones throughout the whole sequence of domains, the attention module A is expected to learn the evolving pattern of feature distributions. Note that we directly introduce learnable statistic vectors ˆ µ 1 , ˆ a 1 and ˆ µ 2 , ˆ a 2 for time points t 1 and t 2 , respectively, since the tokens of historical statistics are not enough for the attention module A to work. That is, the attention module A is only used at time points t t , t ≥ 3 .

Multi-scale Attention Module (MSAM). The above details how to model the evolving pattern using single-scale attention. However, it is limited to learn the pattern using a sliding time window of length 1 over the statistic tokens. Sometimes, the evolutionary pattern may be better captured by considering longer time intervals. For instance, the data of a season in each time window would be more suitable for capturing the evolving pattern of seasonal climate, rather than data of a single day in each time window. Considering this, we introduce multi-scale attention in Fig. 1, which leverages information from observation windows of different lengths to better model the evolving pattern.

Specifically, for the multi-head self-attention module A w responsible for the time window of length w , we slide the time window with stride 1 over historical t -1 statistics tokens and concatenate the statistic tokens in each window to obtain the input ¨ S t -1 w for the attention module A w at time point t t :

And similarly, we use the average output ˘ s t -w +1 w of the attention module A w as the generated statistics for the sliding time window at the next time point, the formulation of which is expressed as

Note that in MSAM, the attention module A w is to predict the statistics in the next sliding time window of length w , instead of one statistic token as in the single-scale attention. This may encourage the attention module to also capture the domain relationships within the window.

Then, we split ˘ s t -w +1 w into w parts: [ ˆ s t -w +1 w , · · · , ˆ s t -1 w , ˆ s t w ] = ˘ s t -w +1 w , where ˆ s j w ∈ R 2 d f can be regarded as the predicted statistic token for time point t j using the attention module A w , j = t -w +1 , · · · , t -1 , t . Finally, the generated statistic token ˆ s t for time point t t using MSAM is denoted as the average of predicted statistic tokens for time point t t at different time window lengths:

where W is the maximum length of the sliding time window. Generally, MSAM integrates evolving patterns learned at different scales, contributing better estimation of future feature distributions.

Feature Standardization. For CDGTD, the second main challenge is generalization. To this end, we leverage the generated feature statistics by MSAM to transform the distribution of corresponding domain into a standard normal distribution, by which the temporal drift is mitigated. Concretely, for feature f t i = E ( x t i ) , its standardized feature z t i via the feature standardization is formulated as

Another benefit of feature standardization is that it allows classifier to be trained on a common feature distribution, thus avoiding the problem of overfitting to current domain and catastrophic forgetting.

3.3 Model Training

This section outlines the four losses L t ce , L t stan , L t con and L t adv involved in the training stage at time point t t . The first one is the essential loss to supervise the model learning for specific tasks, e.g., the following cross-entropy loss L t ce for classification tasks in this paper.

where CE ( · , · ) is the cross-entropy. Here, stopping gradient sg( · ) is adopted to stabilize training.

Losses L t stan and L t,w con are responsible for the training of the attention module A w at time point t t , w = 1 , 2 , · · · , W . The former loss L t stan in Eq. 10 is minimized to ensure that the standardized feature z t i follows a standard normal distribution. Specifically, we first calculate the mean vector and variance vector of the standardized features as

Then the loss L t stan at time point t t is expressed as

where d f is the dimension of features and w is the length of the sliding time window. Here, we just want the learnable/generated ˆ µ t , ˆ a t to approach the statistics of true feature distribution, so stopping gradient operation sg( · ) is conducted on f t i to prevent the gradient of L t stan flowing to the feature encoder E . Otherwise, the gradient of L t stan may distort the feature distribution. Ideally, if L t stan is minimized, the generated statistics ˆ µ t and exp( ˆ a t ) by our MSAM would be equal to the mean vector and standard deviation vector of the true feature distribution of D t . This means that our MSAM can predict future feature distributions based on historical feature statistics.

And the loss L t con is to ensure the consistency between generated statistic tokens in the next sliding time window and real statistic tokens. Specifically, we calculate L t,w con for attention module A w as

where ˆ s j w and s j w , j = t -w +1 , · · · , t -1 are respectively the predicted statistic token for time point t j using the attention module A w and the real statistic token from memory pool M . This loss is expected to also capture the evolutionary pattern within the time window.

The fourth loss L t adv is introduced with this consideration that the feature extractor E may overfit to current domain and lacks generalizability. To avoid this, we additionally conduct adversarial training between the feature encoder E and a discriminator D . Nevertheless, historical data are unavailable. So we instead resort to the learned/generated statistics at previous time points. Concretely, we randomly sample a batch of B samples { f ′ m 1 , · · · , f ′ m B } from each distribution N ( µ m , (exp( a m )) 2 ) , m = 1 , 2 , · · · , t -1 , via Eq. 12 at each iteration, and use them as the proxy of historical domains.

where α is a truncation hyper-parameter to control the sampling area. Then discriminator D is trained to distinguish { f t i } N t i =1 from { f ′ m 1 , · · · , f ′ m B } t -1 m =1 and the feature extractor tries to confuse D . Such adversarial process is achieved by the following loss L t adv :

In practice, the gradient reverse layer (GRL) [15] is used to achieve the adversarial training. To sum up, at time point t t , the model is trained to minimize the following total loss L t total :

For fMoW, backbone DenseNet-121 is too big to apply full GI and DRAIN. So we apply DRAIN only to the classifier and apply GI without the fine-tuning stage.

where λ is a hyper-parameter to balance the loss tradeoff. Once the training procedure at time point t t is finished, we store the learned/generated statistics of current domain D t into memory pool M by

In the inference stage, we use MSAM to generate future statistics ˆ s T + k based on the statistics { s t } T + k -1 t =1 in memory pool M and store it into M as Eq. 15 for the generation at next time point. Due to space limitation, the training and testing procedures are provided in the appendix.

4 Experiments

4.1 Experimental Setup

Thanks to the work in [56], several real-world datasets with distribution shifts over time have been available. And we evaluate EvoS on three image classification datasets ( Yearbook and fMoW from [56] and RMNIST ) and two text classification datasets ( Huffpost and Arxiv from [56]). Yearbook collects data from 1930 to 2013, where we treat every four years as a domain and use the first 16 domains for training ( T = 16 ), the last 5 domains for testing ( K = 5 ). fMoW includes data of 16 years and we set T = 13 , K = 3 . RMNIST contains 9 domains with T = 6 , K = 3 . Huffpost includes data of 7 years with T = 4 , K = 3 . Arxiv collects data of 172 categories for 16 years, with T = 9 , K = 7 . For each training domain of all datasets, we randomly select 90% data as training split and 10% data as validation split. Following the backbones in [56], we use a 4-layer convolutional network [56] for Yearbook, DenseNet-121 [18] pretrained on ImageNet for fMoW, pretrained DistilBERT base model [48] for Huffpost and Arxiv, and the ConvNet in [44] for RMNIST. For each attention module A w in MSAM, its dimension of head d h is set to 8 and its number of heads is set to w · n h . Specifically, n h is set to 16 for Yearbook, 32 for RMNIST, 64 for fMoW and 128 for Huffpost and Arxiv. For optimization, we use the Adam optimizer with lr = 1 e -3 for Yearbook and RMNIST, lr = 2 e -4 for fMoW and lr = 2 e -5 for Huffpost and Arxiv. The batch size is set to 64 for all datasets. As for hyper-parameters, we select them via grid search using the validation splits of training domains and finally use α = 2 . 0 for RMNIST, α = 1 . 0 for others, λ = 1 . 0 , W = 3 for all datasets. For all tasks, we report the mean of 3 random trials. Due to space limitations, please refer to appendix for more details. Code is available at https://github.com/BIT-DA/EvoS .

4.2 Main Results

In addition to the incremental training scenario, we also provide results in non-incremental scenario with all source domains simultaneously available, which serves as upper bounds. Among compared baselines, 'Offline' denotes merging all source domains into one domain and training the model with the merged domain, while 'IncFinetune' represents incrementally training the model in a domainby-domain fashion. For each dataset, we report the accuracy on the nearest target domain ( D T +1 ),

For Huffpost and Arxiv, backbone DistilBERT-base is too big to apply the full GI and DRAIN. So we apply DRAIN only to the classifier and apply GI without the fine-tuning stage.

the average and worst accuracy of future K domains ('OOD avg.': 1 K ∑ K k =1 Acc ( D T + k ) , 'OOD worst": min k ∈{ 1 , 2 , ··· ,K } Acc ( D T + k ) ).

Results on image classification tasks are provided in Table 5, where we can observe that EvoS consistently achieves superior performance across three reported metrics on Yearbook and RMNIST datasets, compared with the methods in incremental training scenario. Especially, EvoS outperforms DRAIN by over 1% according the metric 'OOD worst' on Yearbook and RMNIST, which means that our EvoS learns more robust evolving patterns. In addition, we notice that conventional DG methods IRM [2] and CORAL [50] perform worse than temporal DG method GI [41] on Yearbook and RMNIST, showing the importance of leveraging evolutionary patterns for generalization over temporal drifts. Though being effective for the small networks used in Yearbook and RMNIST, the advantage of GI disappears on fMoW dataset with backbone DenseNet-121, where its firstorder Taylor expansion requires extremely huge computing resources (over 80G). And our available resources cannot afford it. When ablating the Taylor expansion, the performance of GI on fMoW is not the best. So GI is hard to expand on larger models and the similar issue also exists in DRAIN and LSSAE. Finally, we find that for dataset fMoW, not only our method performs unsatisfactorily but also other methods, except for the 'offline' method (i.e., merging all source domain into one domain to train the model). We infer this may be due to the temporal drift is not well presented in this dataset.

Results on text classification tasks are shown in Table 6, where EvoS consistently achieves the best performances according to the average and worst accuracy of future K domains under the CDGTD setting. This can be owed to the more robust evolving patterns captured by our multi-scale attention module (MSAM). Similarly, IRM and CORAL show obvious performance drop compared with DRAIN, once again demonstrating the necessity to learn evolving patterns for the problem of CDGTD. And the large performance gap between continual learning methods (i.e., SI and A-GEM) and our method EvoS on dataset Arxiv verifies that our method EvoS fully utilizes the historical knowledge to learn evolutionary patterns, while SI and A-GEM do not consider this.

4.3 Analytical Experiments

Ablation Study. Firstly, we study the influence of the stopping gradient sg( · ) in the losses L t ce and L t stan . The results of variant A and B in Table 3 almost degenerate to random predictions, compared with variant C. This suggests that using sg( · ) in loss L t stan is essential. Otherwise, the gradient of L t stan would largely distort the learning of feature encoder, causing training collapse. Meanwhile, the inferior performance of variant C to EvoS also indicates that the gradient of L t ce should not interfere with the learning of MSAM. Secondly, we ablate the losses L t adv and L t,w con to testify their necessity. We can see that the results of

EvoS † denotes that we randomly select a domain distribution from the memory pool M to sample a batch of B features for participating in the adversarial training.

17

18

19

20

21

Domain

0.3

0.4

0.5

0.6

0.7

2-norm of

Mean

Ours

Compared

17

18

19

20

21

Domain

0.2

0.3

0.4

0.5

2-norm of

Ours

Compared

1

2

4

8

16

32

64

128

256

Number of heads

nh

90.0

92.5

95.0

97.5

100.0

OOD avg. accuracy (%)

1

2

3

4

5

6

7

8

9

10

Value of scale

W

90.0

92.5

95.0

97.5

100.0

OOD avg. accuracy (%)

(a) Discrepancy between real and generated statistics.

(b) Effect of the number of heads and scale.

Figure 2: (a): The l -2 norm of the difference ∆ between the statistics calculated on each test domain and the statistics generated by MSAM on Yearbook. 'Ours' refers to the way in Eq. 10 and 'Compared' refers to minimizing the l -2 norm of ∆ between statistics calculated in each batch and that generated by MSAM. (b): Effects of using different numbers of heads and scales on Yearbook.

variant D, E and F are all inferior to EvoS. Such results verify two points. One is that the adversarial training conduces to better generalization and mitigating the problem of overfitting. And the other is that the consistency between generated statistic tokens in the window and real statistic tokens in the memory pool imposes stronger constraint to attention module A w , helping to model the evolving pattern more accurately. Thirdly, we testify the influence of truncation when sampling features from normal distributions. The worse result of variant G than EvoS may be due to the sampled outliers hinder the learning of a shared feature space. Finally, the results of variant H and EvoS compare single-scale and multi-scale attention. The better result of EvoS shows the superiority of MSAM.

Last but not least, in Table 4, we investigate the effect of randomly selecting a historical domain distribution vs using all historical domain distributions in the adversarial training. From the results, we see that randomly selecting one historical domain distribution performs worse. This may be because the feature space to be aligned frequently changes if using this manner, making the optimization challenging. By contrast, it is more stable to simultaneously leverage all the preserved domain distributions in the memory pool M in each iteration for the adversarial training.

Choices of L t stan . In addition to training A w via loss L t stan in Eq. 10, we also try another way. Concretely, we directly minimize the l -2 norm of the difference between statistics calculated in each batch and that generated by MSAM, i.e., L t stan = ‖ 1 B ∑ B i =1 sg( f t i ) -ˆ µ t ‖ 2 + ‖ 1 B -1 ∑ B i =1 ( sg ( f t i ) -

1 B ∑ B j =1 sg( f t j )) 2 -( ˆ σ t ) 2 ‖ 2 . Fig. 2(a) plots the discrepancy between generated and real statistics on each test domain of Yearbook when using the two ways to train the attention module. We see that the way used in Sec. 3.3 yields a smaller discrepancy between generated and real statistics, implying that the learned evolving pattern is more accurate. So we choose to use it throughout the experiments.

Effect of Number of Heads and Scale. We further investigate the effect of number of heads and scale of MSAM in Fig. 2(b). Varying the number of heads, the performance exhibits an inverted V-shaped trend, because too few heads would hamper the learning capability of the attention module while too many heads could lead to overfitting. As for difference scales (i.e., the maximum length of sliding time window), we find that multi-scales ( W ≥ 1 ) yields better performance than single-scale ( W = 1 ), showing the superiority of MSAM. Yet, more scales do not always produce better results. One possible explanation is that an excessively large scale relative to the domain sequence length leads to an inadequate number of sliding windows, hampering the learning of attention module.

Visualization of Decision Boundary. In this qualitative experiment, we visualize the change of decison boundary on the inter-twinning moons 2D problem in a gradual manner. Concretely, the 2-Moons dataset in [41] is used, where 10 domains (0 to 9) are obtained by rotating data points counter-clockwise in units of 18 ◦ . In Figure 3, the model is sequentially trained using EvoS until the t -th domain is finished, and then we visualize the decision boundary on current domain D t and the next future domain D t +1 . From the results, we can observe that the decision boundary successfully adapts to future domains, showing that EvoS can truly capture the underlying temporal drift of data.

(a)

t

=5

(b)

t

=6

(c)

t

=7

(d)

t

=8

t

,

y

=

1

t

,

y

= +1

t

+1 ,

y

=

1

t

+1 ,

y

= +1

(a) t-SNE visualization of test domains.

(b) Effect of pool length

L

.

0.1

0.5 1.0 1.5

2.0

2.5

3.0

0.1

5.0

10.0

15.0

20.0

OOD Avg. Acc. (%)

50

60

70

80

90

100

60

70

80

90

(c) Sensitivity of

α, λ

.

Effect of Memory Pool Length. In the main experiments, we do not restrict the size of the memory pool M , since the datasets used in our paper has a moderate number of domains and the memory cost is small. Nevertheless, a fixed memory pool size is more practical when considering a lifelong process, i.e., T →∞ . Thus, we additionally conduct the experiment where the memory pool M is implemented as a FIFO queue with different fixed length L . That is, only the statistics for up to the L most recent historical domains can be stored. The results on Yearbook in Fig. 4(b) demonstrate that our method generally performs well and a relatively large memory pool length is better.

t-SNE Visualization of Standardized Features. To verify our method appropriate for generalization, we visualize the standardized features of all target domains via t-SNE [38] on RMNIST and Yearbook in Fig. 4(a). It can be observed that the standardized features of target domains are well aligned, suggesting that MASM can capture evolving patterns effectively and feature standardization helps address temporal drift appropriately. These enable EvoS to achieve generalization on future domains.

Hyper-parameter Sensitivity. α and λ control the truncation range and the tradeoff the adversarial loss L adv , respectively. Fig. 4(c) shows the sensitivity of EvoS to them on Yearbook, where α ∈ { 0 . 1 , 0 . 5 , 1 . 0 , 1 . 5 , 2 . 0 , 2 . 5 , 3 . 0 } and λ ∈ { 0 . 1 , 0 . 5 , 1 . 0 , 1 . 5 , 2 . 0 , 5 . 0 , 10 . 0 , 15 . 0 , 20 . 0 } . We see that EvoS is more sensitive to λ than α , and large λ values significantly worsen performance. In practice, we suggest selecting λ from a range ≤ 1. For α , its effect is modest and α = 1 generally works well.

5 Conclusion

This paper considers a more challenging problem of continual domain generalization over temporal drift (CDGTD) than conventional DG, where the model is incrementally trained with sequential domains and is required to generalize to unseen domains that are not too far into the future. To this end, we propose an Evolving Standardization (EvoS) method to learn the evolving pattern of sequential domains over the temporal drift and hope to achieve the generalization by transforming the domain distribution into a common normal distribution via feature standardization. Experiments on real-world datasets including images and texts verify the efficacy of EvoS. Since existing DG works focus on conventional setting, we hope this work can encourage more research works on CDGTD.

References

R. Aljundi, M. Lin, B. Goujaud, and Y. Bengio. Gradient based sample selection for online continual learning. In NeurIPS , pages 11816-11825, 2019.

M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv:1907.02893 , 2019.

G. Bai, C. Ling, and L. Zhao. Temporal domain generalization with drift-aware dynamic neural networks. In ICLR , 2023.

S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan. A theory of learning from different domains. Mach. Learn. , 79(1-2):151-175, 2010.

S. Ben-David, J. Blitzer, K. Crammer, and F. Pereira. Analysis of representations for domain adaptation. In NeurIPS , pages 137-144, 2006.

G. Blanchard, G. Lee, and C. Scott. Generalizing from several related classification tasks to a new unlabeled sample. In NeurIPS , pages 2178-2186, 2011.

G. Blanchard, G. Lee, and C. Scott. Generalizing from several related classification tasks to a new unlabeled sample, 2011.

M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS , 2020.

F. M. Castro, M. J. Marín-Jiménez, N. Guil, C. Schmid, and K. Alahari. End-to-end incremental learning. In ECCV , pages 233-248, 2018.

A. Chaudhry, M. Ranzato, M. Rohrbach, and M. Elhoseiny. Efficient lifelong learning with A-GEM. In ICLR , 2019.

T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton. A simple framework for contrastive learning of visual representations. In ICML , pages 1597-1607, 2020.

X. Chen, W. Chen, V. Dinavahi, Y. Liu, and J. Feng. Short-term load forecasting and associated weather variables prediction using resnet-lstm based deep learning. IEEE Access , 11:5393-5405, 2023.

L. Deng. The MNIST database of handwritten digit images for machine learning research [best of the web]. SPM , 29(6):141-142, 2012.

S. Ebrahimi, F. Meier, R. Calandra, T. Darrell, and M. Rohrbach. Adversarial continual learning. In ECCV , volume 12356, pages 386-402, 2020.

Y. Ganin and V. Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML , pages 1180-1189, 2015.

S. Ginosar, K. Rakelly, S. Sachs, B. Yin, and A. A. Efros. A century of portraits: A visual historical record of american high school yearbooks. In ICCV Workshops , pages 652-658, 2015.

R. Gong, W. Li, Y. Chen, and L. Van Gool. Dlow: Domain flow for adaptation and generalization. In CVPR , 2020.

G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In CVPR , pages 2261-2269, 2017.

Y. Iwasawa and Y. Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. In NeurIPS , pages 2427-2440, 2021.

Y. Iwasawa and Y. Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. In NeurIPS , pages 2427-2440, 2021.

P. Izmailov, D. Podoprikhin, T. Garipov, D. P. Vetrov, and A. G. Wilson. Averaging weights leads to wider optima and better generalization. In UAI , pages 876-885, 2018.

J. Kirkpatrick, R. Pascanu, N. C. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell. Overcoming catastrophic forgetting in neural networks. arXiv:1612.00796 , 2016.

A. Kumar, T. Ma, and P. Liang. Understanding self-training for gradual domain adaptation. In ICML , pages 5468-5479, 2020.

M. D. Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. G. Slabaugh, and T. Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. TPAMI , 44(7):3366-3385, 2022.

D. Li, Y. Yang, Y. Song, and T. M. Hospedales. Deeper, broader and artier domain generalization. In ICCV , pages 5543-5551, 2017.

D. Li, Y. Yang, Y. Song, and T. M. Hospedales. Learning to generalize: Meta-learning for domain generalization. In AAAI , pages 3490-3497, 2018.

H. Li, S. J. Pan, S. Wang, and A. C. Kot. Domain generalization with adversarial feature learning. In CVPR , pages 5400-5409, 2018.

S. Li, C. H. Liu, Q. Lin, Q. Wen, L. Su, G. Huang, and Z. Ding. Deep residual correction network for partial domain adaptation. TPAMI , 43(7):2329-2344, 2021.

S. Li, C. H. Liu, B. Xie, L. Su, Z. Ding, and G. Huang. Joint adversarial domain adaptation. In ACM MM , pages 729-737, 2019.

Z. Li and D. Hoiem. Learning without forgetting. TPAMI , 40(12):2935-2947, 2018.

T. Lin, Y. Wang, X. Liu, and X. Qiu. A survey of transformers. AI Open , 3:111-132, 2022.

A. Liu, Y.-C. Liu, Y.-Y. Yeh, and Y.-C. Wang. A unified feature disentangler for multi-domain image translation and manipulation, 2018.

H. Liu, M. Long, J. Wang, and Y. Wang. Learning to adapt to evolving domains, 2020.

M. Long, Z. Cao, J. Wang, and M. I. Jordan. Conditional adversarial domain adaptation. In NeurIPS , pages 1647-1657, 2018.

F. Lv, J. Liang, K. Gong, S. Li, C. H. Liu, H. Li, D. Liu, and G. Wang. Pareto domain adaptation. In NeurIPS , pages 12917-12929, 2021.

F. Lv, J. Liang, S. Li, B. Zang, C. H. Liu, Z. Wang, and D. Liu. Causality inspired representation learning for domain generalization. In CVPR , pages 8036-8046, 2022.

F. Lv, J. Liang, S. Li, J. Zhang, and D. Liu. Improving generalization with domain convex game. In CVPR , pages 24315-24324, 2023.

L. v. d. Maaten and G. Hinton. Visualizing data using t-sne. JMLR , 9:2579-2605, 2008.

K. Muandet, D. Balduzzi, and B. Schölkopf. Domain generalization via invariant feature representation. In ICML , pages 10-18, 2013.

Z. Nado, S. Padhy, D. Sculley, A. D'Amour, B. Lakshminarayanan, and J. Snoek. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv:2006.10963 , 2020.

A. Nasery, S. Thakur, V. Piratla, A. De, and S. Sarawagi. Training for the future: A simple gradient interpolation loss to generalize along time. In NeurIPS , pages 19198-19209, 2021.

A. Nguyen, T. Tran, Y. Gal, and A. Baydin. Domain invariant representation learning with domain density transformations, 2021.

G. Ortiz-Jiménez, M. E. Gheche, E. Simou, H. P. Maretic, and P. Frossard. CDOT: continuous domain adaptation using optimal transport. arXiv:1909.11448 , 2019.

T. Qin, S. Wang, and H. Li. Generalizing to evolving domains with latent structure-aware sequential autoencoder. In ICML , pages 18062-18082, 2022.

J. Rajasegaran, M. Hayat, S. H. Khan, F. S. Khan, and L. Shao. Random path selection for incremental learning. arXiv:906.01120 , 2019.

S. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert. icarl: Incremental classifier and representation learning. In CVPR , pages 5533-5542, 2017.

G. Saha and K. Roy. Continual learning with scaled gradient projection. In AAAI , pages 9677-9685, 2023.

V. Sanh, L. Debut, J. Chaumond, and T. Wolf. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv:1910.01108 , 2019.

S. Schneider, E. Rusak, L. Eck, O. Bringmann, W. Brendel, and M. Bethge. Improving robustness against common corruptions by covariate shift adaptation. In NeurIPS , 2020.

B. Sun and K. Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV Workshops , pages 443-450, 2016.

R. Taori, A. Dave, V. Shankar, N. Carlini, B. Recht, and L. Schmidt. Measuring robustness to natural distribution shifts in image classification. In NeurIPS , 2020.

R. Tiwari, K. Killamsetty, R. K. Iyer, and P. Shenoy. GCR: gradient coreset based replay buffer selection for continual learning. In CVPR , pages 99-108, 2022.

J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In IROS , 2017.

D. Wang, E. Shelhamer, S. Liu, B. A. Olshausen, and T. Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR , 2021.

H. Wang, H. He, and D. Katabi. Continuously indexed domain adaptation. In ICML , volume 119, pages 9898-9907, 2020.

H. Yao, C. Choi, B. Cao, Y. Lee, P. W. Koh, and C. Finn. Wild-time: A benchmark of in-the-wild distribution shift over time. In NeurIPS , 2022.

H. Yao, Y. Wang, S. Li, L. Zhang, W. Liang, J. Zou, and C. Finn. Improving out-of-distribution robustness via selective augmentation. In ICML , pages 25407-25437, 2022.

L. Yuan, B. Xie, and S. Li. Robust test-time adaptation in dynamic scenarios. In CVPR , pages 15922-15932, 2023.

Q. Zeng, W. Wang, F. Zhou, C. Ling, and B. Wang. Foresee what you will learn: Data augmentation for domain generalization in non-stationary environments. arXiv:2301.07845 , 2023.

F. Zenke, B. Poole, and S. Ganguli. Continual learning through synaptic intelligence. In ICML , pages 3987-3995, 2017.

H. Zhang, M. Cissé, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR , 2018.

K. Zhou, Z. Liu, Y. Qiao, T. Xiang, and C. C. Loy. Domain generalization: A survey. TPAMI , 45(4):4396-4415, 2023.

K. Zhou, Y. Yang, Y. Qiao, and T. Xiang. Domain generalization with mixstyle. In ICLR , 2021.

Appendix Contents

Broader Impacts & Limitations

Ethics Statement & Licenses

Algorithm of EvoS

Experimental Setup Details

Experimental Results with Error Bars

Complexity Analysis

Additional Results

A Broader Impacts & Limitations

Our work focus on the problem of continual domain generalization over temporal drift (CDGTD), which aims to generalize the model to unseen future domains by leveraging underlying evolutionary patterns. The effectiveness of our method on several real-world datasets means that it may potentially benefit relevant applications and communities that deal with temporal drifts, e.g., advertisement recommendation, autonomous driving, popularity forecast of media content, etc. Nevertheless, we should also be cautious about possible failures of our method when encountering sudden distribution shifts. In the future, we may explore the automatic identification of domains with severe distribution shifts, allowing us to proactively reject them and mitigate the risk of severe accidents.

B Ethics Statement & Licenses

All the datasets used in the paper are publicly available and are only intended to compare the performances of different algorithms on classification tasks, adhering to the following licenses of these datasets:

Yearbook: MIT licensed

MNIST: CC BY-SA 3.0

fMoW: The Functional Map of the World Challenge Public License

Huffpost: CC0: Public Domain

•

arXiv: CC0: Public Domain

C Algorithm of EvoS

The training and inference procedure for EvoS is provided in Algorithm 1 and 2, respectively.

D Experimental Setup Details

D.1 Dataset Description

Yearbook used in our paper is from [56]. It is based on the Portraits [16] dataset (MIT license), containing the 32 × 32 grayscale images of yearbook portraits from 128 American high schools in 27 states. The data spanning from 1930 to 2013 reflects the evolving fashion styles and shifting social norms throughout the decades. The task is the binary classification of genders. Note that it is only used for comparing the generalization performances of different algorithms on classification tasks. For this dataset, we treat every four years as a domain, resulting in 16 domains. Table 11 provides the number of samples in each domain. The first 16 domains are used for training ( T = 16 ), the last 5 domains for testing ( K = 5 ).

RMNIST is a variant of the MNIST [13] dataset, which comprises 9 domains generated by applying the rotations with degree of 0 ◦ , 10 ◦ , · · · , 80 ◦ in order to MNIST to simulate the temporal drift. The task is to classify a 28 × 28 grayscale digit image from 0 to 9. We use the first 6 domains for training ( T = 6 ) and the last 3 for testing ( K = 3 ).

Algorithm 1: Training procedure for EvoS

fMoW used in our paper is from [56]. It collects the 224 × 224 RGB satellite images from 2002 to 2017 over 200 countries, where the visual features present in satellite data undergo changes over time due to both human and environmental activities. The task to classify the functional purpose of the buildings or the land in the images into one of 62 categories. For this dataset, we treat every year as a domain and Table 12 provides the number of samples in each domain. The first 13 domains are used for training ( T = 13 ) and the last 3 domains are used for testing ( K = 3) .

Huffpost in [56] comprises news headlines from the Huffington Post from 2012 to 2018, the task of which is to classify the news headline into one of 11 news categories ('Black Voices', 'Business', 'Comedy', 'Crime', 'Entertainment', 'Impact', 'Queer Voices', 'Science', 'Sports', 'Tech', 'Travel'). The data spanning for 2012 to 2018 presents changes in the content or style of news along the time dimension. For this dataset, we use the first 4 years for training ( T = 4 ) and the last 3 years for testing ( K = 3 ). The number of samples in each domain is provided in Table 13.

Arxiv in [56] contains paper titles and their corresponding primary categories spanning from 2007 to 2022. The content of Arxiv preprints evolves over time, reflecting the dynamic nature of research fields. And the task is to classify a research paper into one of 172 categories based on its title. We use the data from the first 9 domains for training ( T = 9 ) and the data from the last 7 years for testing ( K = 7 ). The number of samples in each domain is presented in Table 14.

For each training domain of all datasets, we randomly select 90% of data as training split and 10% of data as validation split. The evaluation of generalization performance is based on the whole data of each test domain.

D.2 Implementation Details

All experiments are implemented via PyTorch and the backbones we use mainly adhere to [56]. Yearbook uses a 4-layer convolutional network from [56] as the backbone. For multi-scale attention module (MSAM), we set the dimension of head as d h = 8 for each attention module A w and the number of heads for A w as w · n h , with n h = 16 , w = 1 , · · · , W . And the Adam optimizer with lr = 1 e -3 is used to optimize the model, where the batch size B is set to 64 and the training epochs of each domain are set to 50. As for the hyper-parameters, we use α = 1 . 0 , λ = 1 . 0 , W = 3 .

RMNIST adopts the ConvNet in [44] as the backbone, and we set the dimension of head as d h = 8 for each attention module A w and the number of heads for A w as w · n h , with n h = 32 , w = 1 , · · · , W . The Adam optimizer with lr = 1 e -3 is adopted for model optimization. The batch size and training epochs of each domain are set to 64 and 50, respectively. And α = 2 . 0 , λ = 1 . 0 , W = 3 is used.

fMoW employ the DenseNet-121 [18] pretrained on ImageNet as the backbone. Besides, we use a bottleneck layer [28] to reduce the feature dimensions into 256, and set the dimension of head as d h = 8 for each attention module A w and the number of heads for A w as w · n h , with n h = 64 , w = 1 , · · · , W . Similarly, the Adam optimizer is used, where the learning rate is set to lr = 2 e -4 . The batch size is set to B = 64 and each training domain is trained for 25 epochs. As for the hyper-parameters, we set α = 1 . 0 , λ = 1 . 0 , W = 3 for fMoW dataset.

Huffpost and Arxiv use the pretrained DistilBERT base model [48] as the backbone. For MSAM, we set the dimension of head as d h = 8 for each attention module A w and the number of heads for A w as w · n h , with n h = 128 , w = 1 , · · · , W . Also the Adam optimizer is used with the learning rate lr = 2 e -5 and batch size B = 64 . And α = 1 . 0 , λ = 1 . 0 , W = 3 is used for the two dataset. For the Huffpost dataset, each training is trained for 50 epochs, while the training epochs of each domain are set to 5 for the Arxiv dataset.

We run each task on a single NVIDIA GeForce RTX 3090 GPU for three random trials. For baselines, we also select their hyper-parameters via the grid search using the validation splits of training domains.

For fMoW, backbone DenseNet-121 is too big to apply full GI and DRAIN. So we apply DRAIN only to the classifier and apply GI without the fine-tuning stage.

For Huffpost and Arxiv, backbone DistilBERT-base is too big to apply the full GI and DRAIN. So we apply DRAIN only to the classifier and apply GI without the fine-tuning stage.

E Experimental Results with Error Bars

In this section, we report the mean and standard derivation (denoted as mean (std) ) for each task, when running with 3 random trials. The results with error bars on Yearbook, RMNIST and fMoW are provided in Table 5, and the error bars on Huffpost and Arxiv are given in Table 6.

F Complexity Analysis

Time Complexity of MSAM. Taking one of the multi-head self-attention module A w in MSAM as an example, we denote d f as the feature dimension of its input tokens, and d h , n h and n i denote the feature dimension of its heads, the number of its heads and the number of its input tokens, respectively. Then the time complexity of A w is O (( n 2 i + n i · d f ) · ( d h · n h )) . Since n i will be no larger than the number of training domains T and d h · n h is usually set to d f in transformers, the time complexity of MSAM can be roughly approximated as O ( W · ( T 2 d f + T · d 2 f )) , where W is the number of multi-head self-attention modules in MSAM. It is roughly equivalent to the time complexity of a single multi-head self-attention layer in conventional transformers multiplied by W .

Memory Complexity of M . Assuming that there are T historical domains and the dimension of statistic vectors is d f , then the memory complexity of the memory pool M is O ( T · d f ) . In practice, after being processed by deep neural networks, the dimension of pooled features is usually much smaller than that of original inputs. Moreover, only two vectors need to be stored per domain. Hence, the memory cost of M is relatively small, compared with sample replay-based CL methods. Concretely, Table 7 provides the memory cost of M and the increment of GPU memory cost for EvoS on Yearbook, RMNIST and fMoW datasets with batch size 64.

Model Complexity. Here, we measure the model complexity by the number of parameters. Specifically, Table 8 presents the model complexity of our method EvoS and other two temporal DG methods LSSAE [44] and DRAIN [3] on Yearbook, RMNIST and fMoW datasets. LSSAE [44] introduces sequential autoencoder to explore the underlying continuous structure in the latent space of deep neural networks, where the complicated VAE and LSTM networks require lots of parameters. And DRAIN [3] needs to encode and decode the entire network parameters, which also requires a great number of parameters for large backbone networks. By contrast, our EvoS is overall less complex than these temporal DG methods, and is more friendly to the relatively large backbone network.

G Additional Results

Results under Eval-Stream Manner. In this part, we additionally provide the results when adopting the evaluation manner of Eval-Stream in [56]. Specifically, Eval-Stream denotes the evaluation with domain stream, i.e., the model is evaluated at each timestamp using the average and worst performance on the next K timestamps. Formally, given a sequence of domains with total length T , the average performance Avg stream and worst performance Worst stream under the strategy of Eval-Stream are defined as Avg stream = 1 T K ∑ T K i =1 1 K ∑ i + K j = i +1 Acc i ( D j ) , Worst stream = 1 T K ∑ T K i =1 min j ∈{ i +1 , ··· ,i + K } Acc i ( D j ) , where Acc i ( D j ) is the accuracy on domain D j when using the model at the i -th timestamp. And Table 9 shows the results on Yearbook and Huffpost datasets when using the Eval-Stream evaluation manner. According to the results, we see that EvoS still outperforms other baselines, showing that our method is better at handling the problem of continual domain generalization over temporal drift.

Results on More Datasets. In addition to the datasets provided in [56], we also run our method on the 2-Moons , ONP and Elec2 datasets used in [3]. Specifically, 2-Moons is a variant of the 2-entangled moons dataset by rotating data counter-clockwise in units of 18 ◦ to construct 10 domains, where the rotation angle is used to simulate the temporal shift. For 2-Moons, we use Adam optimizer and a MLP with two hidden layers of hidden size 64 and 128, and B = 64 , lr = 1 e -3 , α = 1 . 0 , λ = 1 . 0 , W = 3 , d h = 8 , n h = 32 , T = 9 , K = 1 . Online News Popularity (ONP) summarizes a heterogeneous set of features about articles published by Mashable in a period of two years. The dataset is split into 6 domains by time and the goal is to predict the number of shares in social networks (popularity). For ONP, we use Adam optimizer and a MLP with one hidden layer of hidden size 128, and B = 64 , lr = 1 e -4 , α = 1 . 0 , λ = 1 . 0 , W = 3 , d h = 8 , n h = 32 , T = 5 , K = 1 . Electrical Demand (Elec2) contains information about the demand of electricity in a particular province. Following [3, 41], the first 30 domains in Elec2 are used (two weeks as one time domain) and the task is to predict whether the demand of electricity in each period (of 30 mins) is higher or lower than the average demand over the last day. For Elec2, we use Adam optimizer and a MLP with two hidden layers of hidden size 128 and 128, and B = 64 , lr = 1 e -4 , α = 1 . 0 , λ = 1 . 0 , W = 3 , d h = 8 , n h = 32 , T = 29 , K = 1 . For these datasets, as in [3, 41], we use the last domain for testing and the rest for training. Please refer to [3] for more dataset details. The experimental results are given in Table 10, where the misclassification errors of compared baselines are all reported from DRAIN [3]. From Table 10, we can observe that our method EvoS still surpasses the most recent method DRAIN, affirming its effectiveness in temporal domain generalization.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMitigating the temporal distribution shift problem is meaningful in realistic. This paper introduces the problem of continual domain generalization over temporal drift (CDGTD), aiming to address the issue of potential temporal distribution shifts in unseen test domains. The authors propose an Evolving Standardization method that utilizes a multi-scale attention module to forecast the distribution of the unseen test domain. They then mitigate the distribution shift by standardizing features using the generated statistics of the test domain. To the best of my knowledge, the method is novel and shows promising results in the experiments section.\n\nSTRENGTHS:\n1. The paper exhibits a commendable structure and offers clear explanations throughout.\n2. The authors conduct ablation studies to examine the individual components of their proposed method.\n3. To the best of my knowledge, the method presented in this paper is novel.\n4. Experiments results confirm the proposed method works well.\n\nWEAKNESSES:\n1. The authors only selected a subset of datasets from the Wilds-Time benchmark for their experiments, it would be valuable to include results on MIMIC-Readmission and MIMIC-Mortality datasets as well, considering wilds-time is the most relevant benchmark for this problem.\n\nQUESTIONS:\n1. Figure 3 (b) shows that the method is not sensitive to hyperparameters \\alpha and \\lambda on yearbook datasets which is pretty strange to me. Do the authors have some intuitions about that? If I am a user using your method, Could the authors provide suggestions or considerations for selecting appropriate values for these hyperparameters? In particular, I am curious about the rationale behind the specific choices of {0.1, 0.5, 1.0, 1.5, 2.0} for the ablation experiment.\n\n2. What is the difference between your experimental settings and wilds-time's eval-stream? Why don't directly follow wilds-time for the experiments?\n\nLIMITATIONS:\nNIL", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a problem formulation for Continual Domain Generalization over Temporal Drift (CDGTD) and proposes the Evolving Standardization (EvoS) method to address the challenge of gradually shifting data distributions over time, aiming to generalize to unseen domains that are not too far into the future. The EvoS method characterizes the evolving pattern of feature distribution and mitigates distribution shift by standardizing features with generated statistics of the corresponding domain. It utilizes a multi-scale attention module (MSAM) to learn the evolving pattern under sliding time windows of varying lengths. MSAM can also generate statistics of the current domain based on the previous domains' statistics and the learned evolving pattern. The paper demonstrates the efficacy of EvoS through experiments on multiple real-world datasets, including images and texts.\n\nSTRENGTHS:\n1. The paper aims to addresses an important problem in machine learning, namely, Continual Domain Generalization over Temporal Drift. The proposed algorithm, EvoS, is a new approach which characterizes the evolving pattern of feature distribution and mitigates the distribution shift by standardizing features with generated statistics of the corresponding domain.\n2. The paper highlights the necessity of learning evolving patterns to address temporal distribution shift.\n3. The paper conducts experiments on multiple real-world datasets, including images and texts, to validate the efficacy of the proposed EvoS method.\n\nWEAKNESSES:\n1. The paper presents a setting where domains arrive sequentially, and models can only access the data of the current domain. However, the authors argue that we should learn from the current source domain to adjust models and generate target domains. This setting contradicts the idea of Domain Generalization, where data from multiple domains may be available simultaneously, or models can access historical data from previous domains.\n\n2. However, the authors of the paper do not provide sufficient context or motivation for the research problem and fail to distinguish this setting from continuous learning and test-time adaptation.\n\n3. The EvoS method introduced in this paper is notably more complex than other baselines. However, the authors do not provide a detailed analysis of the computational requirements of the proposed method.\n\n4. EvoS outperforms other baselines in its tailor-made tasks, which make the efficacy of EvoS unconvincing.\n\nQUESTIONS:\nPlease refer to the weakness section.\n\nLIMITATIONS:\nPlease refer to the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the problem of continual domain Generalization over Temporal Drift (CDGTD), where the domain distribution gradually changes over time and the model needs to generalize to new domains in the near future with training domains sequentially arriving. And this paper also proposes an Evolving Standardization (EvoS) approach to predict and adapt the data distribution of future domains by utilizing a multi-scale attention module to generate the mean and standard deviation of features in current domain and conducting feature standardization based on generated statistics to mitigate the domain shift. Extensive experiments on both image recognition and text classification are done to validate the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. The introduction of the problem of continual domain generalization over temporal drift (CDGTD) is innovative and meaningful. Unlike existing temporal domain generalization approaches, CDGTD incorporates incremental scenarios that mimic real-world situations where training domains arrive sequentially. This assumption adds an extra challenge to generalization, as the model needs to mitigate catastrophic forgetting during the continual learning process.\n\n2. The proposed method EvoS is novel to me. Especially, the inclusion of multi-scale attention module to generate evolved feature statistics is intriguing, where sliding time windows of different lengths are cleverly used to integrate multiscale information for better characterization of evolving patterns.\n\n3. Overall, the quality of the paper is commendable. The authors have conducted a thorough ablation study to examine the impact of various components, and they have also made the code and datasets available in the supplementary material.\n\n4. The paper is mostly well-organized and well-written.\n\n5. The experimental results on image recognition and text classification datasets in the CDGTD setting demonstrate state-of-the-art generalization performance across a diverse range of tasks.\n\nWEAKNESSES:\n1. I am wondering whether the number of heads (n_h) and the feature dimension of heads (d_h) are the same in each attention module A_w, since the authors just simply describe the value of n_h and d_h in MSAM. Yet, there are multiple attention modules in MSAM and the input dimension of each attention module is different. More details are necessary to make it clearer.\n\n2. In Eq. (12) and (13), it seems that the random sampling process is only performed once at the beginning of each training phase. Wouldn't this reduce the diversity and representativeness of the proxy samples for historical domains?\n\n3. As the number of historical domains increases, will the adversarial training suffer from the imbalance problem? What if we randomly select one historical domain in each iteration to participate in the adversarial training?\n\n4. The specific training way of “Offline” and “IncFinetune” in Table 1 and 2 should be provided.\n\nQUESTIONS:\nMinor issue, typos:\nLine 299: Finally, the results of variant H and EvoS compares ...\nLine 309 and 319: MASM -> MSAM\nLine 321: \\beta and \\lambda controls ...\n\nLIMITATIONS:\nLimitations of the proposed approach have been discussed in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers domain generalization over temporal-drift data where the model is trained online and required to generalize to the unseen future domain. The proposed method, Evolving Standardization (EvoS), assumes each domain follows a Gaussian distribution and utilizes transformers to capture the temporal drift by predicting the Gaussian mean and variance of the current domain given historical domains’ means and variances. Specifically, a multi-scale attention module (MSAM) is proposed, which utilizes a sliding window to aggregate domain statistics within a given time scope and makes the transformer predict based on the aggregated domains. The Gaussian statistics generator (namely, feature encoder) is shared over domains and trained in an adversarial way to achieve domain-invariant property. The proposed method is tested on several real-world temporal DG datasets and achieved good performance.\n\nSTRENGTHS:\n1. The proposed problem, which seems like a combination of evolving/temporal domain generalization and online continual learning, is interesting and novel.\n\n2. The proposed method is shown to achieve optimal performance over various existing methods.\n\n3. The paper is in general clearly written and easy to follow.\n\nWEAKNESSES:\n1. The technical contribution is incremental. The single-scale attention is a review of transformer models and domain-invariant adversarial loss is a common tool in either DA/DG works. The key idea of the multi-scale attention module, which is a sliding window-based aggregation of multiple domains’ statistics, is relatively straightforward. Probably some theoretical analyses such as the generalization error bound of the proposed method on the future domain can strengthen the overall technical contribution.\n\n2. The usage of the memory pool is similar to episodic memory in continual learning replay-based methods, and I am wondering about the memory complexity/cost, especially considering the case where the number of temporal domains is large. Also, in online continua learning episodic memory is typically assumed fixed and small to guarantee practical interests, and I am curious if a similar assumption is made in this work.\n\n3. The baselines from the continual learning domain used in this paper are not SOTA (the latest one is A-GEM from 2019), and I encourage the author to compare them with more recent SOTA CL methods. In addition, continual domain adaptation is also a closely related area to this paper, and some methods such as CIDA [1] and CDOT [2] are also interesting to explore.\n\n4. I would encourage the author to consider visualizing the trained model in a gradual manner, such as the visualization of the decision boundary on the Rotated 2-Moons dataset in papers of e.g., CIDA, GI, and DRAIN. Such visualization can serve as qualitative analysis and better demonstrate if the method can truly capture the underlying temporal drift of data.\n\n[1] Wang, Hao, Hao He, and Dina Katabi. \"Continuously indexed domain adaptation.\" Proceedings of the 37th International Conference on Machine Learning. 2020.\n\n[2] Jimenez, G. O., Gheche, M. E., Simou, E., Maretic, H. P., and Frossard, P. Cdot: Continuous domain adaptation using optimal transport. arXiv preprint arXiv:1909.11448, 2019.\n\nQUESTIONS:\nPlease refer to my “Weaknesses” section for the questions.\n\nMy final score will largely depend on the rebuttal and discussion with other reviewers. I am willing to increase my score if my questions are adequately addressed.\n\nLIMITATIONS:\nAs far as I have checked, I did not find potential negative societal impacts in this work. I have listed my concerns in the \"Weaknesses\" section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}