Temporal Domain Generalization with Drift-Aware Dynamic Neural Networks
Authors: Guangji Bai
Chen Ling
Liang Zhao
Venue: ICLR 2023 notable top 5% (2023)
Keywords: Domain Generalization
Sequential Learning Model
Dynamic Neural Network

Abstract: Temporal domain generalization is a promising yet extremely challenging area where the goal is to learn models under temporally changing data distributions and generalize to unseen data distributions following the trends of the change. The advancement of this area is challenged by: 1) characterizing data distribution drift and its impacts on models, 2) expressiveness in tracking the model dynamics, and 3) theoretical guarantee on the performance. To address them, we propose a Temporal Domain Generalization with Drift-Aware Dynamic Neural Network (DRAIN) framework. Specifically, we formulate the problem into a Bayesian framework that jointly models the relation between data and model dynamics. We then build a recurrent graph generation scenario to characterize the dynamic graph-structured neural networks learned across different time points. It captures the temporal drift of model parameters and data distributions and can predict models in the future without the presence of future data. In addition, we explore theoretical guarantees of the model performance under the challenging temporal DG setting and provide theoretical analysis, including uncertainty and generalization error. Finally, extensive experiments on several real-world benchmarks with temporal drift demonstrate the proposed method’s effectiveness and efficiency.

Published as a conference paper at ICLR 2023
TEMPORAL DOMAIN GENERALIZATION WITH DRIFT- AWARE DYNAMIC NEURAL NETWORKS
Guangji Bai; Chen Ling" & Liang Zhao Department of Computer Science
Emory University, Atlanta, GA, USA
ABSTRACT
Temporal domain generalization is a promising yet extremely challenging area where the goal is to learn models under temporally changing data distributions and generalize unseen data distributions following the trends of the change. The advancement of this area is challenged by: 1) characterizing data distribution drift and its impacts on models, 2) expressiveness in tracking the model dynamics. and 3) theoretical guarantee on the performance. To address them, we propose a Temporal Domain Generalization with Drift-Aware Dynamic Neural Network (DRAIN) framework. Specifically, we formulate the problem into a Bayesian framework that jointly models the relation between data and model dynamics. We then build a recurrent graph generation scenario to characterize the dynamic graph-structured neural networks learned across different time points. captures the temporal drift of model parameters and data distributions and can predict mod- els in the future without the presence of future data. In addition, we explore the- oretical guarantees of the model performance under the challenging temporal DG setting and provide theoretical analysis, including uncertainty and generalization error. Finally, extensive experiments several real-world benchmarks with tem- poral drift demonstrate the proposed method's effectiveness and efficiency.
INTRODUCTION
In machine learning, researchers often assume that training and test data follow the same distribu- tion for the trained model to work on test data with some generalizability. However, in reality, this assumption usually cannot be satisfied, and when we cannot make sure the trained model is always applied in the same domain where it was trained. This motivates Domain Adaptation (DA) which builds the bridge between source and target domains by characterizing the transformation between the data from these domains (Ben-David et 2010 Ganin et al. (016;) Tzeng et al., (017). How- ever, in more challenging situations when target domain data is unavailable (e.g., no data from an unknown area, no data from the future, etc.), we need a more realistic scenario named Domain Generalization (DG) (Shankar et al. 2018 Arjovsky et al. 2019 Dou et al. 2019).
Most existing works in DG focus on generalization among domains with categorical indices, such as generalizing the trained model from one dataset (c.g., MNIST (LeCun et al. 1998)) to another (c.g.. SVHN (Netzer et al.) 201T)), from task (c.g., image classification (Krizhevsky al. 2 to another (e.g., image segmentation (Lin et al. 2014)), etc. However, in many real-world applications, the "boundary" among different domains is unavailable and difficult to detect, leading to concept drift across the domains. For example, when a bank leverages model to predict whether a person will be "defaulted borrower". features like "annual incoming". "profession type", and "marital status" are considered. However, due to the temporal change of the society, how these feature values indicate the prediction output should change accordingly following some trends that could be predicted somehow in range of time. Figure I shows another example, seasonal flu prediction via Twitter data which evolves each year in many aspects. For example, monthly active users are increasing, new friendships are formed, the age distribution is shifting under some trends, etc. Such temporal change in data distribution gradually outdated the models. Correspondingly, suppose there was an ideal, always update-to-date model, then the model parameters should gradually change
"Equal contribution. (guangji.bai, ling.ling) @emory.edu Corresponning author. liang.zhao @ emory.edu

---

Published as a conference paper at ICLR 2023
?
users M tweets
users 60 tweets 110 # Avg. following 370
# users 120 M tweets
users 170 tweets 130 y
users 320 tweets 200 B Aug. following 570
users 400 tweets 2211 it Aug. fallowing 620 A.Le
100 B
120 B
Aug. tollowing 350
Avg. following 390
Avg. following 430
-...
....
time
2010
2011
2012
2013
2022
2023 (Future)
Figure 1: An illustrative example of temporal domain generalization. Consider training a model for some classification tasks based on the annual Twitter dataset such that the trained model can generalize to the future domains (e.g., 2023). The temporal drift of data distribution can influence the prediction model such as the rotation of the decision boundary in this case.
correspondingly to counter the trend of data distribution shifting across time. It can also "predict" what the model parameters should look like in an arbitrary (not too far) future time point. This requires the power of temporal domain generalization.
However, as an extension of traditional DG, temporal DG is extremely challenging yet promising. Existing DG methods that treat the domain indices as a categorical variable may not be suitable for temporal DG as they require the domain boundary as apriori to learn the mapping from source to target domains (Muandet et al.] [2013] Mottian et al.] [017 Balaji et al. 20188 (Arjovsyy et al.) 2019). Until now, temporal domain indices have been well explored only in DA (Hoffman et al. 2014
et al.) 2019; Wang et al.) 2020) but not DG. There are very few existing works in temporal DG due to its big challenges. One relevant work is Sequential Learning Domain Gen- eralization (S-MLDG) (Li et al. 2020) that proposed DG framework over sequential domains via meta-learning (Finn et al. 2017). S-MLDG meta-trains the target model on all possible permutations of source domains, with one source domain left for meta-test. However, S-MLDG in fact still treats domain index as a categorical variable, and the method was only tested on categorical DG dataset. A more recent paper called Gradient Interpolation (GI) (Nasery et al. (02T) proposes a temporal DG algorithm to encourage model to learn functions that can extrapolate to the near future by supervis- ing the first-order Taylor expansion of the learned function. However, GI has very limited power in characterizing model dynamics because it can only learn how the activation function changes along time while making all the remaining parameters fixed across time.
The advancement of temporal domain generalization is challenged by several critical bottlenecks, including 1) Difficulty in characterizing the data distribution drift and its influences on models. Modeling the temporally evolving distributions requires making the model time-sensitive. Intuitive ways include feeding the time as an input feature to the model, which is well deemed simple yet problematic as it discards the other features' dependency founding factors changed along time (Wang et al., 2020). Another possible way is to make the model parameters function of time. However, these ways cannot generalize the model future data as long as the whole model's dynamics and data dynamics are not holistically modeled. 2) Lack of expressiveness in tracking the model dynamics. Nowadays, complex tasks have wit- nessed the success of big complex models (e.g., large CNNs (Dosovitskiy et al., 2020)), where the neurons and model parameters are connected as complex graph structure. However, they also sig- nificantly challenge tracking their model dynamics in temporal DG. An expressive model dynamics characterization and prediction requires mapping data dynamics to model dynamics and hence the graph dynamics of model parameters across time. This is a highly open problem, especially for the temporal DG area. 3) Difficulty in theoretical guarantee on the performance. While there are fruitful theoretical analyses on machine learning problems under the independent and identically distributed assumptions (He & Tao 2020). similar analyses meet substantial hurdles to be extended to nut-of--istributt (OOD) problem due to the distribution drift over temporally evolving domains. Therefore, it is essential to enhance the theoretical analyses on the model capacity and theoretical relation among different temporal domain generalization models.
time and dependency on other
To address all the above challenges, we propose a Temporal Domain Generalization with DRift- A Aware dynamIc neural Networks (DRAIN) framework that solves all challenges above simultane-
2

---

Published as a conference paper at ICLR 2023
ously. Specifically, we propose a generic framework to formulate temporal domain generalization by a Bayesian treatment that jointly models the relation between data and model dynamics. To in- stantiate the Bayesian framework, recurrent graph generation scenario is established to encode and decode the dynamic graph-structured neural networks learned across different timestamps. Such a scenario can achieve fully time-sensitive model and can be trained in end-to-end manner. It captures the temporal drift of model parameters and data distributions, and can predict the models in the future without the presence of future data.
We develop a novel and adaptive temporal domain generalization framework that can be trained in an end-to-end manner. 2) We innovatively treat the model as a dynamic graph and leverage graph generation techniques to achieve fully time-sensitive model. 3) We propose to use the sequential model to learn the temporal drift adaptively and leverage the learned sequential pattern to predict the model status on the future domain. 4) We provide theoretical analysis on both uncertainty quantification and generalization error of the proposed method. 5) We demonstrate our model's efficacy and superiority with extensive experiments.
Our contributions include: 1)
RELATED WORK
Continuous Domain Adaptation. Domain Adaptation (DA) has received great attention from re- searchers in the past decade (Ben-David et al., 2010 Ganin et al. 2016 Tzeng et al. 2017) and readers may refer to (Wang& & Deng 2018) for comprehensive survey. Under the big umbrella of DA, continuous domain adaptation considers the problem of adapting to target domains where the domain index is a continuous variable (temporal DA is special case when the domain index is ID). Approaches to tackling such problems can be broadly classified into three categories: (1) biasing the training loss towards future data via transportation of past data (Hoffman et al. 2014]
Ortiz-limenez et al. 2019). (2) using time-sensitive network parameters and explicitly controlling their evolution along time (Kumagai& & Iwata 2016 2017 Mancini et al. 2019), (3) learning repre- sentations that are time-invariant using adversarial methods (Wang et al. 2020). The first category augments the training data, the second category reparameterizes the model, and the third category redesigns the training objective. However, data may not be available for the target domain, or it may not be possible to adapt the base model, thus requiring Domain Generalization.
Domain Generalization (DG). A diversity of DG methods have been proposed in recent years (Muandet et al.) 2013]] Motiian et al.) 2017, Li et al. [017a] Balaji et al. 2018 Dou et al., 2019 Nasery et al. 2021 Yu et al., (022). According to (Wang et al., 2021), existing DG methods can be categorized into the following three groups, namely: (1) Data manipulation: This category of methods focuses manipulating the inputs assist in learning general representations. There r
two kinds of popular techniques along this line: a). Data augmentation (Tobin et al. 2017] (Trembaay let al. 2018). which is mainly based on augmentation, randomization, and transformation of input data; b). Data generation (Liu et al. 2018 Qiao et (002)). which generates diverse samples to help generalization. (2) Representation learning: This category of methods is the most popular in do- main generalization. There are two representative techniques: a). Domain-invariant representation learning (Ganin et al.) 2016 Gong et al., 2019). which performs kernel, adversarial training, explic- itly features alignment between domains, or invariant risk minimization to learn domain-invariant representations; b). Feature disentanglement (Li et al. 2017b), which tries to disentangle the fea- tures into domain-shared or domain-specific parts for better generalization. (3) Learning strategy:
This category of methods focuses on exploiting the general learning strategy to promote the gener- alization capability, e.g. ensemble learning (Mancini et al. 2018), meta-learning (Dou et al. 2019).
gradient operation (Huang et al. 2020), etc.
Existing works above consider generalization across categorical domains, while in this paper, assume the domain index set is across time (namely, temporal), and the domain shifts smoothly over time. Unfortunately, there is only very little work under this setting. The first work called Sequential Learning Domain Generalization (S-MLDG) (Li et al.) (0200) proposed a DG framework over sequential domains based on the idea of meta-learning. A more recent work called Gradient Interpolation (GI) (Nasery et al. (021)) proposes a temporal DG algorithm to encourage model to learn functions that can extrapolate well to the near future by supervising the first-order Taylor expansion of the learned function. However, neither work can adaptively learn the temporal drift across the domains while keeping the strong expressiveness of the learned model.
3

---

Published as a conference paper at ICLR 2023
Training
Test
D
Predict
Enooding Function
Decoding Hunction
Decoding Function R
Encod ng Funetion Ci
Decoding Function Fz
Encoding Function G.
Decoding Function P.
11 LSTM unit
LSTM fa
Jo
unit fg
Forward propagation
Backward propegation
Decision boundary
Skip connection
Figure 2: A high-level overview of our DRAIN framework. Best viewed in color.
3
METHODOLOGY
In this section, we first provide the problem formulation of temporal domain generalization and then introduce our proposed framework, followed by our theoretical analyses.
3.1 PROBLEM FORMULATION
Temporal Domain Generalization. We consider prediction tasks where the data distribution
evolves with time. During training, we are given 7 observed source domains D1, D2, ... Dr sam- pled from distributions on arbitrary time points 1 S t2 S ... S lr, with each 2>, {(x} Mi (*)
N, X, xV, 1=1 1.2, ... ,T where x(") (e) and N. denotes the input feature, label and sample size timestamp fss respectively, and X', V/ denotes the input feature space and label space at times- tamp fs, respectively. The trained model will only be tested on some target domain in the future, i.e., DT+1 where tT+1 2 tr. Our setting further assumes the existence of concept drift across different domains, i... the domain distribution is changing across time by following some patterns.
Our goal to build model that proactively captures the concept drift. Given labeled data from the source domains D1, D2. Dr, we learn the mapping function gw, X. V's each domain Ds. & 1, ,T where wa denotes the function parameters at timestamp La, respectively, and then predict the dynamics across the parameters w1, w2, ... ,W.. Finally, we predict the parameters WT+1 for the mapping function 9wr++1 +1 + V1+1 on the unseen future domain. As shown in Figure due to the temporal drift in data distribution, e.g. the input features such Twitter user age distribution and number of tweets increase each year, the prediction model is expected to evolve accordingly, e.g. the magnitude of model parameter weights will decrease annually. Despite the necessity, handling the above problem is an open research area due to several existing challenges: 1) Difficulty in characterizing data distribution drift well how it influences the model. 2) Lack of expressiveness in automatically capturing the dynamics of how neural network evolves across time. 3) Theoretical guarantee on model's performance (e.g., generalization error, uncertainty) on future domains is hard to obtain due the unknown and (potentially) complicated concept drift.
3.2 PROPOSED METHOD
In this section, we introduce how we address the challenges mentioned above. For the first chal- lenge, we build systematic Bayesian probability framework represent the concept drift over the domains, which instantly differentiates our work from all existing methods in DG. For the second challenge, we propose modeling neural network with changing parameters achieving temporal DG framework that can be trained end-to-end by graph generation techniques. We further improve the proposed method's generalization ability by introducing skip connection module over different domains. Finally, to handle the last challenge, we explore theoretical guar- antees of model performance under the challenging temporal DG setting and provide theoretical analyses of our proposed method, such as uncertainty quantification and generalization error.
dynamic graph and
3.2. A PROBABILISTIC VIEW OF CONCEPT DRIFT IN TEMPORAL DOMAIN
To perform domain generalization over temporally indexed domains, we need to capture the concept drift within given time interval. From probabilistic point of view, for each domain D.,
4

---

Published as a conference paper at ICLR 2023
1, .T, can learn neural network gw. by maximizing the conditional probability (r.,, where w, denotes the status of model parameters at timestamp 4s. Due to the evolving distribution of Dx, the conditional probability Pr(ws, (D.) will change over time accordingly. Our ultimate goal is to predict wT+1 given all training data Di, D2, .. Dr (Du.r for short), i.e., (P1T)) By the Law of total probability, we have
Pr (WT+1 DI:T) = S (WT++ WI.T.
Pr (Wi:r) DI:T) dwi.T.
(I)
inference
training
where Q is the space for WIrr- The first term in the integral represents the inference phase, i.e., how we predict the status of the target neural network in the future (namely, wr+1) given all history statuses, while the second term denotes the training phase, ie.., how we leverage all source domains' training data DI.T to obtaín the status of the neural network on each source domain, namely wI.r. By the chain rule of probability, we can further decompose the training phase as follows:
Pr (WIT PI.T)
II
==1 Pr (wss Wiss-1, DI.T)
T
= Pr (w)) DI)- Pr (w2 w1, (1:2) Pr (wT "I.T-1,D11)>
Here we assume for each time point tss the model parameter wa only depends on the current and previous domains (namely, {D; : i S s}), and there is no access to future data (even unlabeled). Now we can break down the whole training process into T 1 steps, where each step corresponds to learning the model parameter on the new domain conditional on parameter statuses from the history domains and training data, i.e., Pr (Ws+1
Wi:s: Dis: Dx+1), , V <
3.2.2 NEURAL NETWORK WITH DYNAMIC PARAMETERS
Since the data distributions change temporally, the parameter w. in gw, needs to be updated accord- ingly to address the temporal drift across the domains. In this work, we consider leveraging dynamic graphs model the temporally evolving neural networks in order retain maximal expressiveness.
Intuitively, a neural network Iu can be represented as an edge-weighter graph G = (V, E, v). where each node V represents neuron of gu while each edge E E corresponds to connection between two neurons in Iw- Moreover, given a connection e between neuron u and v, i.e., e = (u, E F, function &b F. + denotes the weight parameter between these two neurons, i.e., (u, u) Wu.o. V (u, v) E E. Essentially, values indexed by all edges in E and w represents the entire set of parameters for neural network g. Notice that we give a general definition of g/ so that both shallow models (namely, linear model) and deep neural networks (e.g.. MLP, CNN, RNN, GNN) can be treated as special cases here. We aim to characterize the potential drift across domains by optimizing and updating the graph structure (i.e., edge weight) of gw. You et al. (2020) have proven that optimizing the graph structure of the neural network could have smaller search space and more smooth optimization procedure than cxhaustively searching over all possible connectivity patterns.
= <bE)= {Wa, (u, e) E E} is set of parameter
We consider the case where the architecture or topology of neural network Iw is given, i.c., V and E' are fixed, while the parameter w is changing constantly w.r. time point t.. In this sense, we can write wa (E(s) where v-(+ss) (abbrev. via) depends only time point ts. Now the triplet G = (V, E, vss) defines a dynamic graph with evolving edge weights.
3.2.3 END-TO-END LEARNING OF CONCEPT DRIFT
Given history statuses ( } of the neural network learned from {Diss), we aim at generalizing and extrapolating w,+1 so that it produces good performance on the new domain Do+1 in an end-to-end manner. In fact, by viewing the neural networks (W1:s) as dynamically evolving graphs, a natural choice is to characterize the latent graph distribution of {wi:s} by learning from its evolving trend. Consequently, w's 's can be directly sampled from the distribution for the prediction in future domains.
We characterize the latent distribution of (w1:.) as sequential learning process based on a re- current architecture, and each unit fo in the recurrent model is parameterized by 0 to generate ws by accounting for previous {w; i s). Specifically, at each recurrent block (i.e., time step) t.. Je produces two outputs It). where m, is the current memory state and hy is a latent proba- bilistic distribution (i.c., hidden output of fo) denoting the information carried from previous time steps. The latent probabilistic distribution by allows us to generate the dynamic graph wa by a de- coding function F-(.). Intuitively, different from existing works that train and regularize neural
5

---

Published as a conference paper at ICLR 2023
network single domain Naseryy et al. 2021). here we focus directly searching for distribu- tion of networks with "good architectures" : Lastly, the sampled w. is encoded by a graph encoding function G,)+), which then serves as the input of next recurrent block. Such recurrent model is trained on a single domain Dz to generate w, for prediction by minimizing the empirical loss, i.e., mine,g.9 مi=1 ((ww, (x(")), v,,))) where f(---) can be cross-entropy for classification or MSE for re- gression. The optimal ws on domain D, will then be fed into the next domain Dz+1 along with the memory state m, input to guide the generation of Ws+1 until the entire training phase is done. For the inference phase, we feed the optimal parameters from the last training domain, namely wr, into the encoding function and leverage the recurrent block, together with the memory state mer to predict the latent vector on the future domain Dr-1, followed by the decoding function to decode the latent vector and generate the optimal parameters WT+1-
3.2.4 LESS FORGETTING AND BETTER GENERALIZATION
During the training of recurrent models, it is also likely to encounter the performance degradation problem. Such a problem can be severe in temporal DG since more complicated concept corre- lation exists between each domain. In addition, if the training procedure on each domain D, takes a large number of iterations to converge, we may also observe the forgetting phenomenon (i.e., the recurrent model fe will gradually focus on the current training domain and have less generalization capability for future domains). To alleviate such a phenomenon, we leverage a straightforward tech- nique skip connection to bridge the training on D. with previous domains (Di.a-1). Specifically,
wss (we-r:--1)) = wa + A -i=s- Wi.
(3)
where À is regularization coefficient and r denotes the size of the sliding window. The skip connec- tion could enforce the generated network parameters to contain part of previous network's ss infor- mation, and the implementation of the fixed-sized sliding window can better alleviate the potential drawback of the computational cost. We summarize the overall generative process in Appendix A.2
3.3 THEORETICAL ANALYSES
In this section, we provide a theoretical analysis of our proposed framework's performance in the target domain. Our analyses include uncertainty quantification and generalization error. Uncertainty characterizes the dispersion or сттоr of an estimate due to the noise in measurements and the finite size of data sets, and smaller uncertainty means less margin of error over the model predictions. On the other hand, generalization error measures how accurate the model's prediction is on unseen data. Our analyses show that our proposed DRAIN achieves both better prediction accuracy as well as smaller margin of error on target domain compared with online and offline DG baselines.
First, we introduce two DG methods, namely online baseline and offline baseline as defined below:
Definition 1. Given timestamp t,+1 and domains D1, D2. Da+1. and model parameter state from previous timestamp, namely ws. Define online model Mon and offline model Man as w,+1 = argmax Pr(ws+1les [ws, D,+1) and wo41 argmax 2, Pr(we++1 (D:s+1), respectively.
Pr(ws+t|PP.stat>
Offline method Mon is trained using ERM over all source domains, while online method Mon considers one-step finetuming over the model parameter on each new domain's 's dataset. Both Man and Mon are time-oblivious, i.c., unaware of the concept drift over time.
Assumption 1. Consider a parameterized function (-(-) to approximate P( wi), which is the unknown ground-truth concept drift of the model parameter distribution. It is assumed that the prior over qe follows normal distribution with: (w)] Var(ya, (w)) = V E Ω.
Definition 2 (Predictive Distribution). Given training sample D1, D2.... Dr and input feature from future domain, namely XT+1, I, the predictive distribution can be defined as
Pr ( XT-1, PL.)) = /Pr Pr (ū XT+1, Pr (WT-1) D1rr)doT11
(4)
Our first theorem below shows that by capturing the concept drift over the sequential domains, our proposed method always achieves the smallest uncertainty in prediction on the future domain.
Theorem 1 (Uncertainty Quantification). Given training domains D1, D2.... , Dr where Var(D,)
is the same, we have the following inequality over each method's predictive uncertainty, ie.. the variance of predictive distribution defined in Eq. 4 Var(Mous)) < <ar(Mou)) S Var(Mog).

---

Published as a conference paper at ICLR 2023
Table Performance comparison of all methods in terms of misclassification error (in %) for classi-
fication tasks and mean absolute error (MAE) for regression tasks (both smaller the better.) Results of comparison methods on all datasets except "Appliance" are reported from Nasery et (2021) "_" denotes that the method could not converge the specific dataset.
Classification (in %)
Regression
Model
2-Moons
Rot-MNIST
ONP
Shuttle
Elec2
House
Appliance
Offline
22,4 + 4.6
18.6 + 4.0
33.8 = 0,6
0.77 + 0.1
23.0 -: 3.1
11.0 + 0.36
10.2 1 1.1
LastDomain
14.9 0.9
17.2 3.1
36,0 0.2
0.91
0.18
25.8 1 0.6
9.1 0.7
IncFinetune
16.7 3.4
10
34.0
0.83 - 0.07
27.3
9.7 0.01
8.9
CDOT
9.3 1.0
1.0
3..1
0.17
17.8
CIDA
10,8 + 1.6
9.3
3.67
14.1 0.2
0.06
8.7 0.2
GI
3.5 1.4
1.3
0.8
0.29
= 0.05
16.9 + 0.7
9.6
8.2 0.6
DRAIN
3.2 1.2
7,5 1.1
38.3 1.2 0.26 + 0.05
12.7
0.8
9.3 0.14
6.4 + 0.4
Our second theorem shows that, besides uncertainty, our proposed method can also achieves smallest generalization error thanks to learning the concept drift.
Definition 3. Given predictive distribution in Eq. as well as ground- truth label WT+1 from the future domain, define the predictive or generalization error as err = ((E (() D1:T) ), FT+1).
Theorem 2 (Generalization Error). Assume (-) has Lipschitz constant with upper bound Lupper and lower bound Llower W.r.l w. We have the following inequality over each method's ss predictive error def defined above: err(Mours) < err(Mon) (rr(Mog).
Complexity Analyses. In our implementation, the encoding and decoding functions are instantiated as MLPs. The total number of parameters of the encoding and decoding functions is O(Nd + C), which is linear in N. Here N is the number of parameters in predictive models (namely w), d the width (i.e., number of neurons) of the last hidden layer of the encoding and decoding functions, and C denotes the number of parameters for all the layers before the last for the encoding and decoding functions. Additionally, in many situations, the first few layers of representation learning could be shared. Hence, we do not need to generate all the parameters in w, but just the last few layers.
EXPERIMENT
In this section, we present the performance of DRAIN against other state-of-the-ar approaches with both quantitative and qualitative analysis. The experiments in this paper performed 64-bit machine with 4-core Intel Xeon W-2123 @ 3.60GHZ, 32GB memory and NVIDIA Quadro RTX 5000. Additional experiment settings and results (e.g.. hyperparameter setting and scalability analysis) are demonstrated in the appendix.
4.1 EXPERIMENT SETTING
Datasets. We compare with the following classification datasets: Rotated Moons (2-Moons), Ro- tated MNIST (Rot-MNIST), Online News Popularity (ONP), Electrical Demand (Elec2), and Shut- tle; and the following regression datasets: House prices dataset (House), Appliances energy predic- tion dataset (Appliance). The first two datasets synthetic, where the rotation angle is used proxy for time. The remaining datasets are real-world datasets with temporally evolving character- istics. Dataset details can be found at Appendix A.I.I
Comparison Methods. We adopt three sets of comparison methods: practical baselines that do not consider the concept drift, including 1). Offline that treats all source domains single domain, 2). LastDomain that only employs the last training domain, and 3). IncFinetune that sequentially trains on each training domain. Continuous domain adaptation methods that focus only on DA. including 1). CDOT (Ortiz-Jimenez et al. 2019) that transports most recent labeled examples to the future, and 2). CIDA (Wang et al.) (2020) that specifically tackles the continuous DA problem; and one temporal domain generalization method: GI (Nasery et al.) 2021).
All experiments are repeated 10 times for each method, and we report the average results and the standard deviation in the following quantitative analysis. More detailed description of each compar- ison method and the parameter setting can be found in Appendix A.1.2 and A.1.3 respectively.
Our open-source code is available httptt:::////........m /ttttss://gtteie http::////gttuuu.com hwtp:///ee..t.etttttteeeeeeeeeee

---

Published as a conference paper at ICLR 2023
A
descd diss
Cane A cods
doss
-1
-5
-65 12
(a) Training Domain D2 (b) Training Domain D4 (c) Training Domain De
(d) Testing Domain Do
Figure 3: Visualization of the decision boundary of DRAIN (blue dots and red stars represent different data classes). As the distribution of data points is consistently changing, shown in Figure
3a 3d DRAIN can effectively characterize such temporal drift and predict accurate decision boundaries on the unseen testing domain in Figure 3d
3.5
2.0
1.5
0.-
1.0
0.5
0.0
0.5
-10-
-15
-1
(a) Offline
(b) LastDDomain
(c) IncFinctune
2.0
dbss B
dass &
dlsss class 8
2.0
diss A daise B
dlass cless
2.0
Class A class B
dass A daes
3.5
1.5
1.0
1.0
1.0
0.5
0.
0.5
3.0
0.0
0.0
2.5
45
4.5
-,00
1.0
4.0-
3.5
15
3.5-
(d) CDOT
(e) CIDA
(1) GI
Figure 4: Visualization of decision boundary (blue dots and red stars represent different data
classes), where the right subfigure of comparison methods Figure 4a 4i demonstrate the decision boundary predicted for the test domain D7+1. the left subfigure Figure 4al shows the decision boundary learned from the all data points in the concatenated training domain ([DI.-.. Dr.), the left subfigure in Figure 4b shows the decision boundary learned from all samples in the last training domain Dy, and the left subfigures in Figure 4c
4f show the decision boundary learned on D4.
4.2 QUANTITATIVE ANALYSIS
We first illustrate the performance of our proposed method against comparison methods. The periments are conducted in both classification and regression tasks with the domain generalization setting, i.e., models are trained on the training domains and deployed on the unseen testing domain.
As can be seen from Table | DRAIN consistently achieves competitive results across most datasets. Specifically, DRAIN excels the second-best approaches Blec2 (CIDA), House (GI) and Appliance (GI) by a great margin. The only exception is the ONP dataset, where the Offline method achieves the best result and all state-of-the-an methods cannot generalize well on unseen testing domains since the ONP dataset does not exhibit a strong concept drift. Additionally, all time-oblivious base- lines perform rather unsatisfactorily since they are not capable of handling the concept drift of the data distribution. Both CDOT and CIDA can generate better results than time-oblivious baselines, yet their generalization ability on the unseen domains is still limited as the maintained time-invariant representation in both methods cannot address the concept drift without any data in the testing do- main. As the only method that addresses the temporal domain generalization problem, GI imposes a gradient regularization with non parameetiii activation function to handle the concept drift, which relies too much on the task-specific heuristic. contrast, DRAIN proposes to sequentially model each domain in end-to-end manner, which could address the concept drift more inherently.
8

---

Published as a conference paper at ICLR 2023
4.3 QUALITATIVE ANALYSIS
We compare different methods qualitatively by visualizing the decision boundary on the 2-Moons dataset. As shown in Figure 3a 3c we demonstrate the decision boundary predicted by DRAIN at D2, D4. D6 training domains, and the final predicted decision boundary on the testing domain Dy (Figure 3d). As can be seen, DRAIN can successfully characterize the concept drift by sequentially modeling the (Dr). and the learned decision boundary could rotate correctly along time.
We further visualize the decision boundary learned by other comparison methods in Figure 4a Firstly, the left subfigure in Figure 4alshows the decision boundary Iearned by the Offline method on the concatenated training domains (P1rr), and the learned decision boundary overfits the training data and shows poor performance when generalizing on the unseen testing domain (the right sub- figure of (a).. Furthermore, as the current state-of-the-art continuous domain adaptation methods, CDOT transports the most recent labeled data points in Dy the future, which makes the learned decision boundary almost temporal-invarian of domain generalization. CIDA utilizes the adversarial training technique to solve the domain adaptation, yet the predicted decision boundary in Figure 4e is less stable than other state-of-the-an methods due to its model complexity. Lastly, even though GI is the only method proposed tackle the temporal domain generalization problem, the produced decision boundaries, shown in both the training domain and testing domain (Figure(fT) are still less accurate than our proposed method, since they heavily utilize heuristics to regularize the gradient.
41
(Figure 4d and cannot generalize well in the scenario
4.4 SENSITIVITY ANALYSIS
We conduct sensitivity analysis on the depth of the neural network 9w. for DRAIN. As shown in Figure 5) the optimal number of hidden layers for Iw, 2 and I 2-Moons and Electric dataset, respectively. The curve on both datasets has an inverse "U" shape,
Average Accuraiyy
95
Awerage Accurncy
85
90
85
2-Moons
80
Elec2
80
meaning that too few layers may limit the general expressiveness of our model, while too many layers could potentially hurt the generalization ability due to overfitting.
Numbar Layer
Layer
Figure 5: Sensitivity analysis on the number of layers of the generated neural network by DRAIN.
4.5 ABLATION STUDY
We further conduct ablation study on three datasets to evaluate the effect of different com- ponents in DRAIN, and the results are exhib- ited in Table 2 Specifically, we remove the se- quential learning model in DRAIN, and the re- sulted ablated model X RNN corresponds to the offline baseline model. We also independently remove the skip connection module to let the sequential learning model uniformly acquire in- formation from all previous domains, and the resulting model named X Skip.C.
Table 2: Ablation study. Comparison of perfor- mance between our method and two alternatives across two datasets for classification tasks and one datasets for regression tasks.
Ablation RNN X Skip.C DRAIN
2-Moons
Rot-MNIST
House
22.4 4.6
19.3 3.4
11.0 = 0.36
7.1 = 1.3
10.3 + 1.7
9.7 0.13
3.2 = 1.2
7.5 1 1.1
9.3 0.14
As shown in the table, yet each component effectively contribute to the overall model perfor- mance, modeling the temporal correlation between all domains by a sequential model can provide a rather larger performance gain. In addition, removing the skip connection in the sequential learning model would make DRAIN hard to capture the long-range temporal dependency among domains since long-range domain information could potentially be forgotten during the model learning.
5 CONCLUSION
We tackle the problem of temporal domain generalization by proposing dynamic neural network framework. We build a Bayesian framework to model the concept drift and treat a neural network as dynamic graph to capture the evolving pattern. We provide theoretical analyses our proposed method, such as uncertainty and generalization error, and extensive empirical results to demonstrate the efficacy and efficiency of method compared with state-of-the-art DA and DG methods.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work supported by the National Science Foundation (NSF) Grant No. 1755850, No. 1841520, No. 2007716, No. 2007976, No. 1942594, No. 1907805, a Jeffress Memorial Trust Award, Amazon Research Award, NVIDIA GPU Grant, and Design Knowledge Company (subcon- tract number:
10827. 01...002.120.06.
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 00093090,, 2019.
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain gen- eralization using metu-regularization. Advances neural information processing systems, 31, 2018.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Machine learning, 79(1):151-177 2010.
Sébastien Bubeck, Yuanzhi Li, and Dheeraj M Nagaraj. A law of robustness for two-layers neural networks. In Conference on Learning Theory, pp. 804-820. PMLR, 2021.
Luis M Candanedo, Véronique Feldheim, and Dominique Deramaix. Data driven prediction models of energy use of appliances in a low-energy house. Energy and huildings, 140:81-97, 2017.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 29(6): 141-):14 2012.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020.
Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generaliza- tion via model-agnostic learning of semantic features. Advances in Neural Information Processing Systems, 32, 2019.
Kelwin Fernandes, Pedro Vinagre, and Paulo Cortez. A proactive intelligent decision support system for predicting the popularity of online news. In Portuguese conference on artificial intelligence, pp. 535-535-50 546. Springer, 2015.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1133 1135. PMLR, 2017.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. The journal of machine learning research, 17(1):006-2000 2030, 2016.
Rui Gong, Wen Li, Yuhua Chen, and Lue Van Gool. Dlow: Domain flow for adaptation and general- ization. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 2477-2486, 2019.
Fengxiang He and Dacheng Tao. Recent advances in deep learning theory. arXiv preprint arXiv:2012.10931 2020.
Judy Hoffman, Trevor Darrell, and Kate Saenko. Continuous manifold based adaptation for evoly- ing visual domains. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 867-874, 2014.
Zeyi Huang, Haohan Wang, Eric Xing, and Dong Huang. Self-challenging improves cross-domain generalization. In European Conference on Computer Vision, pp. 124-140. Springer, 2020.
10

---

Published as a conference paper at ICLR 2023
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Atsutoshi Kumagai and Tomoharu Iwata. Learning future classifiers without additional data. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
Atsutoshi Kumagai and Tomoharu Iwata. Learning non-linear dynamics of decision boundaries for maintaining classification performance. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-basee learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 1998.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp- 5542-5550, 2017a.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy Hospedales. Sequential learning for domain generalization. In European Conference on Computer Vision, pp. 603-619. Springer, 2020.
Wen Li, Zheng Xu, Dong Xu, Dengxin Dai, and Lue Van Gool. Domain generalization and adapta- tion using low rank exemplar svms. IEEE transactions on pattern analysis and machine intelli- gence, 406)::114-1127, 2017b.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Alexander H Liu, Yen-Cheng Liu, Yu-Yo Ying Yeh, and Yu-Chiang Frank Wang. A unified feature dis- entangler for multi-domain image translation and manipulation. Advances in neural information processing systems, 31, 2018.
Massimiliano Mancini, Samuel Rota Bulo, Barbara Caputo, and Elisa Ricci. Best sources forward: domain generalization through source-specifie nets. In 2018 25th IEEE international conference on image processing (ICIP), pp. 1353-1357. IEEE, 2018.
Massimiliano Mancini, Samuel Rota Bulo, Barbara Caputo, and Elisa Ricci. Adagraph: Unifying predictive and continuous domain adaptation through graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 0666-6577, 2019.
Saeid Motiian, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto. Unified deep super- vised domain adaptation and generalization. Proceedings of the IEEE international conference computer vision, pp. 5715-5725, 2017.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference Machine Learning, pp. 10-18. PMLR, 2013.
Anshul Nasery, Soumyadeep Thakur, Vihari Piratla, Abir De, and Sunita Sarawagi. Training for the future: A simple gradient interpolation loss to generalize along time. Advances in Neural Information Processing Systems, 34, 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Guillermo Ortiz-Jimenez, Mireille Gheche, Effrosyni Simou, Hermina Petric Maretic, and Pas- cal Frossard. Cdot: Continuous domain adaptation using optimal transport. arXiv preprint arXiv: 2019.
Fengchun Qiao, Long Zhao, and Xi Peng. Learning to leam single domain generalization. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12556- 12565, 2020.
11

---

Published as a conference paper at ICLR 2023
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross-gradient training. arXiv preprint arXiv:1804.1074 10745, 2018.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, Varun Jampani, Cem Anil, Thang To, Eric Cameracei, Shaad Boochoon, and Stan Birchfield. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 969-977, 2018.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference computer vision and paltern recognition, pp. 7167-7176, 2017.
Hao Wang. Hao He, and Dina Katabi. Continuously indexed domain adaptation. arXiv preprint arXiv:2007.01807, 2020.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Wenjun Zeng, and Tao Qin. Generalizing to unseen domains: A survey domain generalization. arXiv preprint arXiv:2103.03097, 2021.
Mei Wang and Weihong Deng. Deep visual domain adaptation: survey. Neurocomputing, 312: 135-153, 2018.
Jiaxuan You, Jure Leskovec, Kaiming He, and Saining Xie. Graph structure of neural networks. In International Conference on Machine Learning, pp. 10881-10891. PMIR, 2020.
Dazhou Yu, Guangji Bai, Yun Li, and Liang Zhao. Deep spatial domain generalization. arXiv preprint arXiv:2210.00729, 2022.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 EXPERIMENTAL DETAILS
A.1.1 DATASET DETAILS
We expand upon the datasets used for experiments in this section. We highlighted the sentence that describes the domain drift within each dataset.
Rotated 2 Moons: This is variant of the 2-entangled moons dataset, with a lower moon and an upper moon labeled 0 and respectively. Each moon consists of 100 instances, and 10 domains are obtained by sampling 200 data points from the 2-Moons distribution, and rotating them counter- clockwise in units of 18". Domains 0 to 8 (both inclusive) are our training domains, and domain 9 is for testing. Each domain obtained by rotating the moons counter-clockwise in units of 18", hence the concept drift means the rolation of the moon-shupe clusters.
This is an adaptation of the popular MNIST digit dataset Deng (002)) where the task is to classify digit from () to 9 given an image of the digit. We generate 5 domains by rotating the images in steps of 15 degrees. To generate the i-th domain, we sample 1 1,000 images from the MNIST dataset and rotate them counter-clockwise by 15x i degrees. We take the first four domains as train domains and the fifth domain as test. Similar to 2-Moons, each domain here is generated by rotating the images of digits by 15", hence the concept drift means the rotation of the images.
Rotated MNIST:
Online News Popularity: This dataset Fernandes et al.) (215)) summarizes heterogeneous set of features about articles published by Mashable in a period two years. The goal is to predict the number of shares in social networks (popularity). We split the dataset by time into 6 domains and use the first 5 for training. The concept drift is reflected in the change of time, but previous works have proven Nasery et al. (207)) the concept drift is not strong.
This dataset provides about 58,000 data points for space shuttles in flight. The task is multiclass classification with heavy class imbalance. The dataset was divided into 8 domains based on the time points associated with points, with times between 30-70 being the train domains and 70 -80 being the test domain.
Shuttle:
Electrical Demand This contains information about the demand of electricity in a particular province The task is, again binary classification, to predict if the demand of electricity in each period (of 30 mins) was higher or lower than the average demand over the last day. We consider two weeks be one time domain, and train on 29 domains while testing on domain 30. Each domain is generated by considering the demand of electricity within certain two weeks, so the domain drift can be regarded as how the electricity demand is changing seasonally.
This dataset has housing price data from 2013-2019. This is regression task to predict the price of a house given the features. We treat each year as separate domain, but also give information about the exact date of purchase to the models. We take data from the year 2019 to be test data and prior data as training. Similar to Elec2, the concept drift in this dataset is how the housing price changed from 2013-2019 for a certain region.
House Prices Dataset:
Appliances Energy Prediction: This dataset Candanedo et al. (0017) is used to create regression models of appliances energy use in a low energy building. The data set is at 10 min for about 4.5 months in 2016, and we treat each half month as single domain, resulting in 9 domains in total. The first 8 domains are used for training and the last one is for testing. Similar to Elec2, the drifi for this dataset corresponds to how the appliances energy usage changes in a low energy building over about 4.5 months in 2016.
1.1 DETAILS OF COMPARISON METHODS
Practical Baseline. 1). Offline: this is a time-oblivious model that is trained using ERM on all the source domains.
2). LastDomain: this is time-oblivious model that is trained using ERM on the last source domains. 3). IncFinetune: we bias the training towards more recent data by applying the Baseline method described above the first time point and then, fine-tuning with reduced learning rate on the subsequent time points in sequential manner. This baseline corresponds to the online model we defined in Definition
13

---

Published as a conference paper at ICLR 2023
Continuous Domain Adaptation Methods. 1). CDOT: this model transports most recent labeled examples D" to the future using a learned coupling from past data, and trains a classifier on them.. 2). CIDA: this method is representative of typical domain erasure methods applied to continuous domain adaptation problems. 3). Adagraph: This method makes the hatch norm parameters time- sensitive and smooths them using given kernel.
Temporal Domain Generalization Method. 1). Gl: this method proposes a training algorithm to encourage model to learn functions which can extrapolate well to the near future by supervising the first order Taylor expansion of the learnt function.
A.1.3 PARAMETER SETTING
We use Adam optimizer for all our experiments, and the learning rate for all datasets are uniformly set to be @ 3.60GHz, 32GB memory and NVIDIA Quadro RTX 5000. We set hyperparameters for each comparison method with respect to the recommendation their original paper, and we specify the architecture as well as other details for each dataset's s experiments as follows.
All experiments are conducted on 64-bit machine with 4-core Intel Xeon W-2123
2-Moons.
The number of layers in the LSTM is set to be 10, and the network architecture of Jus consists of 2 hidden layers, with a dimension of each. We use ReLU layer after each hidden layer and Sigmoid layer after the output layer. The learning rate is set to be 1e- 4.
Rot-MNIST. The number of layers in the LSTM is set to be 10, and the network architecture of 9w consists of 2 convolution layers with kernel shape 3 x 3, and each convolution layer is followed by max pooling layer with kernel size and stride 2. The latent representation then transformed by two linear layers with dimensions 256 and 10. We use ReLU layer after each hidden layer and a Sigmoid layer after the output layer. The learning rate is set be le
ONP. The number of layers in the LSTM is set to be 10, and the network architecture of gu. consists of 2 hidden layers with bias terms, and the dimensions of each layer are 20. We use ReLU layer after each hidden layer and a Sigmoid layer after the output layer. The learning rate is set to be
Shuttle.
The number of layers in the LSTM is set to be 5, and the network architecture of gw; consists of 3 hidden layers with bias terms, and the dimensions of each layer 128. We use ReLU layer after each hidden layer and a Sigmoid layer after the output layer. The learning rate is set to be 5.
Elec2. The number of layers in the LSTM is set to be 10, and the network architecture of go: consists of 2 hidden layers with bias terms, and the dimensions of each layer are 128. We use ReLU layer after each hidden layer and a Sigmoid layer after the output layer. The learning rate is set to be 5.
The number of layers in the LSTM is set to be 10, and the network architecture of gla consists of 2 hidden layers with bias terms, and the dimensions of each layer are 128. We use ReLU layer after each hidden layer and no activation layer after the output layer. The learning rate is set to be le- - 5.
House.
Appliance. The number of layers in the LSTM is set to be 10, and the network architecture of gw, consists of 2 hidden layers with bias terms, and the dimensions of each layer are 128. We use ReLU layer after each hidden layer and no activation layer after the output layer. The learning rate is set to be le 5.
A.1.4 TRAINING TIME ANALYSIS
Table 3: Comparison of training time (seconds) between our method and two baselines across two datasets for classification tasks and one datasets for regression tasks.
Model
2-Moons
Elec2 154.3
Appliance
CIDA
45.2 + 0.87
2.1
287.5 = 2.6 189.3 = 2.1
GI
19.3 + 0.43
136.4 = 1.9
DRAIN (ours)
15.4 + 0.37
99.2 1.3
170.3 = 1.8
14

---

Published as a conference paper at ICLR 2023
We further conduct the model scalability analysis by comparing the running time of proposed method with two other state-of-the-ar art baselines: GI and CIDA on three datasets (i... 2-Moons, Elec2, and Appliance). As shown in Table our proposed method can generally achieve the shortest training time among the three methods. However, we notice that GI is relatively slower in the total running time due to the model pretraining and finetuning step, and the low efficiency in CIDA is due to the expensive computation cost for training GAN. Compared to these approaches, DRAIN only consists of one sequential learning model address the data distribution drift in the end-to-end manner, which could achieve generally better performance while attaining its efficiency.
A.1.5 SCALABILITY OF NUMBER OF DOMAINS
The time complexity of our framework with respect to the number of domains is linear (equivalent to the complexity of the recurrent neural network with respect to the input sequence length). The number domains can only affect the total training time since we need to iteratively feed in new domain to train the proposed recurrent model.
We conduct the following experiment to support our argument. We create synthetic datasets with 10, 100, and 1000 domains, each of which has two labels with 10 training instances. We follow the parameter setting in the 2-Moons dataset (the exact parameter setting can be found in Appendix A.3), and their runtime is demonstrated in the following table.
Table 4: Scalability of DRAIN for number of training domains.
Number of domains Running time
10
2.66
100
28.51
1000
292.49
A.1.6 IMPORTANT REMARKS
In this section, we provide some important remarks over the proposed DRAIN framework.
Graph generation can handle large graphs and there are a number of existing works that can handle large graphs. Our model is general framework that can choose different graph generation methods as needed.
Neural networks are networks (i.e., graphs) of neurons, which have gained lots of research interest in recent years. Recent research (e.g.. (You et al.) (0200) have found that the per- formance of neural network architectures is highly correlated with certain graph charac- teristics. In this work, aim at characterizing the potential drift across the domains by optimizing and updating the graph structure of the neural network because optimizing the graph structure of a neural network has been proven have a smaller search space and more smooth optimization procedure than exhaustively searching over all possible connec- tivity patterns. Last but not least, our approach allows the entire neural network/model to change across time, which in turn maximizes our model's expressiveness.
A.1.7 ENLARGED DECISION BOUNDARY FIGURES OF GI AND DRAIN
class
class
-2
-2
-1
0
(a) GI
(b) DRAIN
Figure 6: Comparison the decision boundary on the future domain of 2-Moons dataset between the state-of-the model GI and the proposed model DRAIN.
15

---

Published as a conference paper at ICLR 2023
Figure 6 is direct comparison of decision boundaries predicted by the state-of-the- art method GI (Figure 6a) and the proposed method DRAIN (Figure 6p). As can be seen from the figure, the decision boundary predicted by DRAIN can consistently classify two classes with few exceptions. the decision boundary predicted by Gl has less confidence (i.e., wider band) in predicting middle points, and few errors also made in predicting points the right side.
A.2 OVERALL GENERATION PROCESS
We summarize the detailed forward propagation of DRAIN as below:
a 0, my Go(z), 2 ~~ (0, 1)
a1 (,((w), w1 Fg(ht), (mi, h1) fo 0o)
"1 Gr w1 Co(I1)) (m1, h1) (1(m1.0))
42 (w2),, w2 = (w (w) }), w2 = Fe(h1), (m2, h2) = amm
42 Gn sw2 (w1) w2 = Fe (e((1)) (1u2, = so(m2, u2)
Gx(22), we = owwr, fwr
= Fe(h1)) (m, h.) = fo (mz,
where each denotes the input of fe. In this work, we utilize LSTM as the recurrent architecture, and fo becomes single LSTM unit. To initialize the whole generative process, we take a random noise z as input for the first domain D1, which is drawn from a standard Gaussian distribution. The initial memory state my is also transformed from a by an initial encoding function Go(-).
A.3 THEORY PROOF
In this section, we provide the formal proof for Theorem I and Theorem 2 in main context.
A.3.1
PROOF FOR THEOREM
Proof. By definition of the predictive distribution,
P(j|x4+1 D1.1)
P(i)x-+n Dirr)dur+1
(5)
= P((lxTH1w7+))) (1 (Prrrr/Pr..
Our goal is to prove that the variance of this predictive distribution for proposed method, online baseline and offline baseline follows the inequality as in Theoren
Ours v.s. Online Baseline
Here we prove that Var(Mouns) < Var(Man).
Notice that the first term on the right hand side of Eq. 5 namely P( (1,41,444)) corresponds to deployment of the model with parameter WT+1 on the future domain Dr+1. hence the variance of 1()+ only depends the noise randomness coming from XT+1 as long wr+1 given. In other words, the uncertainty coming from P( can be cancelled for both methods since
considering the same set of domains. Now the problem reduces prove that the variance of the second and third terms on the right hand side of Eq. 5 for our model is smaller than those for the online baseline.
Notice that
((Wir
-/ P(wi P(wz|w1, (o) P(0 06) P(wzlwa, 01) 0(0))
(6)
... P(WTTlTTT1 Dr, 0T) P(0 OT 2000
where 0 is the parameter of the parameterized function to approximate the ground-trutt truth drift of w, as defined in Assumption For example, P(w] ) denotes that train the model the very first
16

---

Published as a conference paper at ICLR 2023
domain and P(w2/w1, D2, D2,00) denotes that continue to train the model on the second domain but with initialization of w2 as 48, (w1) where wi is learned from the previous domain and 4% is trying drift between w2 and w1, i.c., P(wy[w1). In our Bayesian
to capture the conditional probability framework, we treat @e as a learnable function (e.g., LSTM unit in our proposed method) and we subscript of differentiate the status of after the training each domain. In other words, go will be updated after the training on each domain (at least for our method). Notice that do always denotes the parameter initialization as in Assumption
By Bayes' rule, we have:
P(we-1we, D++1, 0--1) P(ge (wi)) P(De+ale+t))
prior on we+1
likelibood
where P(go,-, (wx)) can be regarded as the prior of wt+1 because as we mentioned 40, denotes the initialization of wi+1 before we train the model on domain Di+1, and P(P corresponds to the likelihood of training w++1 on D++1- In addition,
P(O (or, w1+1, 0c-1) (P(00))) P(ax,we+1) (0x)
xx PP(0022) P( w0-1)) P(ux, w4+1
(8)
P(Oo)
II
P(ws (1+1)
prior on
likelihood
for any 1,2,3,... T In the equation above, this time the prior over parameter and wis Wi+1 can be regarded as the "training data" for 0;.
For the online baseline, since it only keeps one-step finetuning of the model and does not learn how We evolves, the Og for the online baseline is always equal to the prior, i.e. 0r 0g. In other words, (wt) = P(9e, (wi) and (00 (et, w1+1, 0t-1) = P(Oo). t for the online baseline.
Since we follow the standard routine and assume all distributions are Gaussian, by Bayesian Theo- rem, we know that the posterior distribution always has variance smaller than the prior distribution under the expectation, i.e.,
[ Far( (0e w1,1, 0t-1)] Var(Oo).
which proves that our method has smaller variance in terms of Eq. 8 On the other hand, since the second term on the right hand side of Eq. is the same for both methods, and for the first term P(ge,_, (wx)), by our Assumption [ we know that for baseline Pr(ge_ (wx) (r())_ variance is basically da.- For our method, after each training step across a new domain our will get updated and achieve smaller variance (because of posterior variance of Gaussian) so we also prove that our method has smaller variance in terms of Eq. 7 Two parts combined prove that our method has smaller variance in the third term of Eq. namely P( (Di.T).
Pr(ge, (wx) so the
The last step is to compare the variance from the second term in Eq. namely P(wr+al (www))))))).).))))))))))))))>))) For online baseline, basically it uses the parameter from the last training domain, i.e., wr the final model on the future domain, i.e., P(wT+1|w11) = ((ge((WT))
On the other hand, for method have P(wT++ ( P(ger (WT)) which has smaller variance due to the posterior variance of Gaussian.
All together finish the proof for Var(Mauns) Var(MMa).
Online Baseline v.s. Offline Baseline
This case is simpler prove. Again, the first term the right hand side of Eq namely P(i)xT=1,,>++) can be cancelled in this case. Moreover, the second term, namely P has the same variance for both baselines, i..., Var (aar(P(w++1l/r) makes sense since two baselines do not learn the drift and the uncertainty in predicting w1 based on wr is always the same the prior distribution of fg-
Var(P(ga, (WT))) on, This
Hence, it suffices to compare the uncertainty of the last term of Eq. namely P(w Recall
Mon
wt41 argmax We+1 P(w4+1 (10,41/w,2000 D(+1)
(10)
Mar
W++1
argmax.... W1+1 P(we+1 (P1.+1)
17

---

Published as a conference paper at ICLR 2023
For offline baseline, are using all dataset far, namely Dia+1 to train the model while the online baseline only uses D++1- Since we are considering domain generalization with temporal concept drift, i.., for each # have D; Dj (otherwise merge them), the randomness of u D; is at least as large as that of Da+1 alone, i.e., Var(U D.) z Var(P:+1).
To prove this, let's consider the case of two domains D1 and D2 without loss of generality. Also, assume the sample size for both domains are equal. By definition of variance, we have
Var(Di) >i=1 (TL. - 21)
Var(D2) = Di= (22,, p2)
(11)
n
n
while
Ei=i (m),) 1 j2 Σi= ( 12
Var(D; UD2)
(12)
2n
where jn and #2 is the sample mean for each domain, respectively and n denotes the sample size. Hence,
(ar( + Var(D2))
EE (x1,1
41 )² + Σイ= (x2,,
41 M1
Σ (01,5 111)2+ + 1 (202,1 1
122)
2n
2n
A1 12 )²
P1 + 12
n (11, -11) + Σ
n
(x2,4 m2)²)
11
222
1=1
1=1
1=1
(11,1
111 + 12)2
+ (x2
111 + 22222
(#1,5 21)² + (x2,t /2)
2
2
1 (M1 + 12))0111 (11 + p2) 211 + 2 +
2 _ 22)² -1/ -1² -²²
(13)
(m1 p2)=1.1 (#1 x2) - 1 2 2222
(m1 p2) 2221) 1 2011 122)²
11222 (m
1/1 (111 - 122)2)
1
(M1 )22)² 2
2 0,
where the equation from the third last row to the second last row is under expectation as [[ x2,1)] = f11 #22 Since we assume Var(D,) Var(P2), finish the proof that Var(D, U D2) 2
Var(D2). One can generalize this conclusion onto three or more domains.
Finally, combining Var(U!++ D2) is at least as large as that of Var(P:41) with Bayes' rule, one can finish the proof.
A.3.2 PROOF OF THEOREM 2
Proof. We finish our proof in two steps. First we prove that the generalization error of our method smaller than that of the online baseline.
By definition, we know that
err = (B P(.r)). (OWT_ (XT+1), VT+1).
(14)
where g., denotes the target neural network with parameter w. and WT+11 denotes the parameter status on the (T 1) domain (i.e., future domain).
18

---

Published as a conference paper at ICLR 2023
For online baseline, since it does not consider the temporal information, the parameters on the future domain will be the same as the parameters after the training on the last source domain, i., for online baseline we have WT+1 wT-
For our method, we have wT+1 = (wr), where qo is the recurrent structure and 0r denotes the parameter status of the recurrent structure after training the first T domains. In other words, our method utilizes the recurrent structure to generate the model parameters on the next domain. Now it suffices to show that
(Dwr+) (XT+1), (xr+)) vr+1) < (Bor (xrr(x
(xT+1), (11+))
(15)
where WT+11 4 (wT). Here, the LHS and RHS above corresponds to the generalization error of method and the online baseline, respectively.
Recall that do represents the LSTM unit in our case, and we train the LSTM unit to approximate the transition probability P(w++1|0xx) i.e., how neural network g's parameter distribution changes over time. From a probabilistic point of view, during training of the LSTM unit go. we basically minimize the empirical loss which is equivalent to
min DKL [0o|| (uoll((we+)))e)). t= 1.2,
(16)
As mentioned in Assumption 1 we denote du as the initialization of go. On the other hand, after --1 1 times of training over the LSTM unit on the T source domains, 0 will converge to an optima denoted as 8* Hence, the model parameter wT+1 generated by the converged LSTM unit for sure will be closer to the ground truth than that generated by the random initialized LSTM unit, i.c.
|1 (WT) qo- (WT) || (WT) - -o- (wT)]
(17)
By Lipschitz continuity of au over the parameter w. we have
Tlower ||ا - w'1l <. ([o)(z) Iw (x)] Tupper || o'Il V Ex
(18)
where is defined as the input space of neural network ga. Bubeck et (202T) proved that the Lipschitz constant actually can have lower bound for neural network.
Denote w" qo- (WT), i.c., the optimal parameter for the target neural network g on the future domain. Then, it directly follows [8. that
1 (x) Dw (x)] Lower |||T| - w=
(19)
I (a) Iw (z)] < Lupper -|WT-I - w'1l-
Denote 1 w ||/||||||++ w"ll. Since neural network Iw is continuous function of w, there always exists a constant 8 > 0 such that, within the sphere centering at w" with radius 8, namely S(w", 6), the local lower and upper bound for the Lipschitz constant of gu could satisfy Lupper/ / Loower < r. The reason behind this is, as ō approaches O, due to the continuity of gu, the upper bound and lower bound of Lipschitz constant within S(w" 6) will become closer and finally identical, i.e., limıg-;n -,o Lupper/ / Liower = 1. On the other hand, by Eq. [7 we know that r is always greater than 1, so is always possible to find a ō to satisfy the above condition. As a result,
Lupper |W0+11 - wI
< + Lupper ||0141- - w0ll < Liower ||w - wI
(20)
Llower |WT - II
Hence, || (z) Iw (x)]l < 11 (x) gw (z)]]- Since gw- (xx) is the optimal neural network on the future domain, Iw- 1)) should achieve the lowest loss as defined Eq. 15 Combined everything above together finishes the first step of our proof.
The second step of our proof is for the comparison between two baselines. We consider the case where the drift of we is monotonic but our proof can be generalized to other cases easily.
As can be shown,
Online baseline: E ((WW441))))) Elge, ([r) (ur))==p) = WT:
(21)
Offline baseline: B[P(wT+1k<1:)) Elgo, (w) (w) -
If we denote distance function over the domains, as d. we assume that d(D Dr41) < d(D,, Dr41). By the monotonic assumption, the distribution of each Di.T is changing along a certain direction. Hence, among them Dr has the distribution most close to that of Dr+1- In other words, the online baseline finetunes the model so its wr is leaning towards the last domain while the offline baseline is using the averaged domains to train the model, which finishes the proof.
19


Reviews:
[{"review_id": "pMH0ZGg8eK", "reviewer": "Reviewer_NDvd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the issue of temporal drift. The proposed methodology consists of a recurrent neural network modelling the parameters of the classification/regression model, assuming constant topology of this later. This recurrent model aims to encapsulate the latent change in parameters through time. The parameters of the network at a new time point are then drawn from this latent state that encodes the concept drift. The model is compared to multiple strategies on a set of both synthetic and real datasets for both regression and classification tasks.\n", "summary_of_the_review": "The paper studies an interesting problem and proposes a novel and efficient solution. However, the theoretical results do not strengthen considerably the paper and I am not sure of their correctness. ", "strength_and_weaknesses": "The paper approaches the important problem of temporal concept drift with an elegant solution. The draft is quite dense but remains clear. The experimentation is thorough with useful ablation studies and interesting discussions and limitations.\n\nThe theoretical results are interesting and show the importance of modelling the concept drift, but do not quite show why the proposed approach should outperform the other state-of-the-art strategies. Importantly, two key claims are unclear:\n- Why should the variance of the union of domains always be larger than the last domain? For instance, if each domain at time $t$ is a gaussian with mean $0$ and variance $t$ (the variance increases with time, meaning that the union has a lower variance than the last observed distribution)\n- Why does the posterior always have a lower variance than the prior? I think this claim only stands in expectation.\nFinally, the formalisation of the concept drift under which the methodology would perform better would be particularly valuable. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "e2aLHIUVHI", "reviewer": "Reviewer_WCLC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the problem of domain generalization where the domain index is across time. The authors proposed a Bayesian framework to explicitly model the concept drift over time, i.e., predicting the updated model on the future domain based on historical domains. A recurrent neural network is employed to learn the evolving dynamics of model parameter distribution based on how data distribution evolves across the domains. To achieve maximal expressiveness, the authors proposed to model the neural network as a dynamic graph which allows all connections between the neurons to adapt temporally. The authors also introduced a technique based on skip-connection to mitigate catastrophic forgetting. Theoretical analyses showed smaller margin of error and uncertainty in prediction of the proposed method compared with two baselines. The proposed method achieved excellent performances on several classification and regression datasets compared with several state-of-the-art domain generalization methods.", "summary_of_the_review": "Overall, this paper makes a non-trivial and meaningful exploration of temporal domain generalization. Both theoretical and empirical results are provided with great performance. The paper is well-written and quite easy to follow. Hence, I would recommend accept of this paper. I am open to hearing from the authors if I made any misunderstandings.", "strength_and_weaknesses": "Strengths:\n\n1.\tThe originality of this paper is great. The problem is clearly defined.\n\n2.\tThe topic of temporal domain generalization is interesting and prompt. \n\n3.\tThe proposed method is both theoretically and technically sound to me.\n\n4.\tHow the authors capture the temporal drift of model parameter distributions by considering the dynamic graph over a recurrent structure is intuitive. This paper makes a non-trivial exploration of the temporal domain generalization problem.\n\n5.\tExtensive theoretical and empirical results round up the good work.\n\nWeaknesses:\n\n1.\tCould the authors elaborate more on the reason of using skip-connection in Sec 3.2.4? In other words, why providing the LSTM unit with previous domain’s information will mitigate the catastrophic forgetting on the future domain?\n\n2.\tSome details are not clear enough. For example, what is the shape of the parameter $\\omega$ into the recurrent unit? In addition, what initialization is used for the model parameters in this paper?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UD31XBrRH1", "reviewer": "Reviewer_vJXs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The Paper studies temporal domain generalization with dynamic neural networks. \nFor a discrete set of datasets: D1, ..., D_T at different timestamps: t_1, ..., t_T, goal is to build a model that captures concept drift. \nIn particular, model learns how to predicts parameters of NN for timestamp T+1 for the mapping X_{T+1}-> T_{T+1}.\n\nAuthors, propose a Bayesian probabilistic framework, that is based on the law of total probability where integration over inference part P(w_{T+1}| w{1:T}, D{1:T}) and training part is done P(w{1:T}| D_{1:T}), in the end2end manner. \n\nThe authors provide two main theorems about uncertainty quantification and generalization error. \n\nAuthors, show on on 5 datasets for classification and 2 dtasets for regressions that they have very competitive performance w.r.t. baselines: offline, lastdomain, CDOT and others. \n\n\n\n", "summary_of_the_review": "Interesting paper but needs more clarity w.r.t. theoretical guarantees. ", "strength_and_weaknesses": "Strength:\n- Interesting problem setting\n- Interesting approach with solid results.\n\nWeaknesses:\n- Paper does not provide enough information to understand the full details (e.g. see comments on integrals and proofs)\n- Proofs and theorems are not fully clear\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%