Weight Diffusion for Future: Learn to Generalize in Non-Stationary Environments
Authors: Mixue Xie
Shuang Li
Binhui Xie
Chi Harold Liu
Jian Liang
Zixun Sun
Ke Feng
Chengwei Zhu
Venue: NeurIPS 2024 poster (2024)
Keywords: domain generalization
evolving pattern
diffuison model
weight generation
domain-incremental
Primary area: other
TL;DR: A weight diffusion approach that leverages the conditional diffusion model to learn the evolving pattern of parameters across domains and further generated customized classifiers for evolving domain generalization in the domain-incremental setting.

Abstract: Enabling deep models to generalize in non-stationary environments is vital for real-world machine learning, as data distributions are often found to continually change. Recently, evolving domain generalization (EDG) has emerged to tackle the domain generalization in a time-varying system, where the domain gradually evolves over time in an underlying continuous structure. Nevertheless, it typically assumes multiple source domains simultaneously ready. It still remains an open problem to address EDG in the domain-incremental setting, where source domains are non-static and arrive sequentially to mimic the evolution of training domains. To this end, we propose Weight Diffusion (W-Diff), a novel framework that utilizes the conditional diffusion model in the parameter space to learn the evolving pattern of classifiers during the domain-incremental training process. Specifically, the diffusion model is conditioned on the classifier weights of different historical domain (regarded as a reference point) and the prototypes of current domain, to learn the evolution from the reference point to the classifier weights of current domain (regarded as the anchor point). In addition, a domain-shared feature encoder is learned by enforcing prediction consistency among multiple classifiers, so as to mitigate the overfitting problem and restrict the evolving pattern to be reflected in the classifier as much as possible. During inference, we adopt the ensemble manner based on a great number of target domain-customized classifiers, which are cheaply obtained via the conditional diffusion model, for robust prediction. Comprehensive experiments on both synthetic and real-world datasets show the superior generalization performance of W-Diff on unseen domains in the future.

Weight Diffusion for Future: Learn to Generalize in Non-Stationary Environments
Mixue Xie Beijing Institute of Technology mxxieObit.eom
Shuang Li -
Binhui Xie
Beijing Institute of Technology shuangli@bis edu. cn
Beijing Institute of Technology
binhuixie@bit.cm
Chi Harold Liu
Jian Liang
Zixun Sun
Beijing Institute of Technology chiliu@bit. edu. cn
Kuaishou Technology
Tencent
liangj 1a030kuu1 shou com
sunzixun8126.0 com
Ke Feng
Chengwei Zhu
Tencent
Tencent
ri enchareeeggteeneent. com
chavezzhu@tencent..cm
Abstract
Enabling deep models to generalize in non-stationary environments is vital for real-world machine learning, data distributions are often found to continually change. Recently, evolving domain generalization (EDG) has emerged to tackle the domain generalization in time-varying system, where the domain gradually evolves over time in an underlying continuous structure. Nevertheless, it typically assumes multiple source domains simultaneously ready. It still remains an open problem to address EDG the domain-incremento setting, where source domains non-static and arrive sequentially mimie the evolution of training domains. To this end, we propose Weight Diffusion (W-Diff), a novel framework that utilizes the conditional diffusion model in the parameter space learn the evolving pattern of classifiers during the comain-incremental training process. Specifically, the diffusion model conditioned on the classifier weights of different historical do- main (regarded as reference point) and the prototypes of current domain, to learn the evolution from the reference point the classifier weights of current domain (regarded as the anchor point). In addition, domain-shared feature encoder is learned by enforcing prediction consistency among multiple classifiers, so as mitigate the overfitting problem and restrict the evolving pattern to be reflected in the classifier much possible. During inference, we adopt the ensemble manner based great number of target domain ccntomiee classifiers, which are cheaply obtained via the conditional diffusion model, for robust prediction. Comprehensive experiments on both synthetic and real-world datasets show the superior generalization performance of W-Diff on unseen domains the future. [
Introduction
Domain generalization (DG) deals with fundamental problem in modern machine learning 13
46
47]. where performance degeneration often occurs when deep models encounter cutt-o-distribution
(OOD) data [38] [56]. The goal of DG is to learn a model that can perform well on unseen target
domains by leveraging labeled data from multiple related but different source domains 23
19
57
22
Corresponding author. Code is available at atattps://gittuuu.comm BAABBIII--DAA-DDA
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Despite of abundant works on DG and promising progress so far, they typically embark upon the generalization among stationary and discrete environments (29). where the distribution shift among domains is obvious and remains static over time. In contrast, there have emerged several works 50 in recent years, where the data
on evolving domain generalization (EDG) [24 29] 51
distribution gradually shifts in an underlying continuous structure, e.g., the age-related structural changes in the optic nerve in ocular diseases [12.. But most of EDG methods still assume multiple source domains simultaneously ready, which may be impractical in real world. As the data distribution constantly evolves along time, training data from new data distributions will continue to emerge. Hence, equipping models with lifelong learning ability is crucial for their practical applications.
Despite of the fact that existing researches on continual learning [16 32 [ have studied the em- powerment of lifelong learning, their focus is on maintaining the performance of seen tasks, instead of generalizing on unseen domains in the future. Therefore, is still an open problem to achieve evolving domain generalization in the domain-incremental setting, where source domains sequentially arrive to mimic the dynamics of training domains. To this end, previous work EvoS 45 models the features from each domain as Gaussian distribution and proposes to capture the evolving pattern at the feature level by leveraging self-attention mechanism to generate the feature mean and variance for future domain based on those of historical domains. However, the assumption that features adhere to a Gaussian distribution may not always be applicable. Different from EvoS, we propose to excavate evolving pattem at the parameter level and further achieve Nomain-customizze parameter generation.
Inspired by neural network diffusion 42 that there exist specific parameter patterns in optimized model layers and these patterns can be modeled with diffusion model, we propose to capitalize on the strong modeling ability of diffusion models to capture the evolving pattern of optimized classifiers across domains. To achieve this, we propose a Weight Diffusion (W-Diff) approach, which is specialized for EDG in the domain-incremmental setting. Specifically, to address the problem of inaccessible historical data, we maintain first-in-firstouu (FIFO) queue to store the optimized classifier weights of historical domains. The stored classifier weights of each historical domain serve reference point to calculate the change between the classifier weights of current domain (regarded as the anchor point) and that of corresponding historical domain. The changes of classifier weights between the anchor point and different reference points provide evolving patterns at different time intervals, which can be utilized to make the modeling of evolving patterns more robust.
In addition, for guidance on how to switch from a reference point to the anchor point, we condition the diffusion model on the class prototypes of current domain along with the reference point. Then, the conditional diffusion model is trained to model the distribution of residual classifier weights, i... the change of classifier weights between the anchor point and reference point. Meanwhile, to reduce the overfitting of the feature encoder and to restrict the evolutionary pattern to be reflected only in the classifier as much as possible, we learn a domain-shared feature space by enforcing the predictions from different classifiers to be consistent. Finally, during the inference stage, adopt weights ensemble to give robust predictions based on a great number of generated classifier weights by the diffusion model that is conditioned current class prototypes and different reference points.
Contributions: 1)
We study the under-explored area of evolving domain generalization in the setting and explore the innovative usage of diffusion model for this problem. 2) We propose novel weight diffusion (W-Diff) approach to capture the evolutionary pattern at the parameter level, orthogonal to previous feature level approaches. Capitalizing the strong generative ability of diffusion model, W-Diff can generate customized parameters by controlling the condition and make robust predictions via weights ensemble. 3) Comprehensive experiments on both synthetic and real-world datasets verify the effectiveness and superiority of W-Diff on generalization.
2 Related Work
Evolving Domain Generalization (EDG) learns the evolving pattern underlying in multiple source
domains to achieve generalization capability on the unseen future target domains over time 22 50
24 52 44 29). To name a few, SDE-EDG (50) introduces stochastic differential equations (SDEs) to model the evolving pattern through individual temporal trajectories. DRAIN (2] builds on the Bayesian framework and leverages LSTM to infer the future status of the whole network. However,
most of these methods, except for ARAI,, require multiple source domains to be simultaneously available. Very recently, EvoS [45 focuses on EDG with sequentially arriving domains, considering
2

---

the low efficiency of training the model from scratch once the accumulated domains are updated. It assumes that features for each domain follow Gaussian distribution and then models the evolving pattern of the feature distribution, while this assumption is not always suitable. Besides, in the generalization process of multiple consecutive target domains, the statisties generated from previous timestamps are used as inputs to the attention mechanism to generate the statistics at next timestamp, which in turn serve as the input for next generation. This manner is likely to cause error accumulation if previous generation is not accurate. Orthogonal to EvoS [45] we proposes to mine the evolving pattern at the model parameter level and further implement domain-oriented parameter generation via controlling the condition of the diffusion model to avoid the potential error accumulation in EvoS.
Continual Learning (CL) focuses on the scenario where the model is trained on a sequence of tasks, and the model is required to adapt to current task and meanwhile maintain the performance 11]. The literature on this field is abundant. Most of the techniques can be categorized into rrchitecture-base (34 [9, representation-based [4] 43 10],
on previous tasks 53 16
this work, we also continually train models on sequential domains, but the goal is to generalize well novel domains in the near future.
regularization-batt [66 53 32 and replay-basee based [5] 39
Parameter Generation has been gaining great interests with the rise of diffusion models 42
54.. For example, p-diff 42) directly generates high-performing neural network parameters from random noises with a standard latent diffusion model, which verifies the feasibility of modeling the parameter distribution via diffusion models. Nevertheless, p-diff uses unconditional diffusion model and can only generate parameters for indistributtio data. G.pt [28] collects the loss, error or return of task model checkpoints during training as the condition for the diffusion model. However, it is designed for single dataset to which the training data belongs, thus struggling with distribution shifts. D2NWG [36] uses CLIP (30] to extract features for each sample and leverages Set Transformer [[]] to generate dataset encoding from these features. Then, the diffusion model is conditioned on the dataset encoding. But labeled training set samples of new dataset are required to obtain the dataset encoding, which is infeasible unlabeled target domains. addition, ProtoDiff (8] and MetaDiff [4] generate prototype classifiers for the meta-test stage by conditioning the diffusion model on the information (c.g., the prototypes in 8 and the gradients in (54)) from the support set. Yet, they concentrate few-shot learning and the support set requires labeled data, which is unavailable in the target domain of EDG. By contrast, we aims at capturing the evolving pattern among source domains and leveraging it to enable generalization on future target domains without any labeled target data.
3 Preliminaries
3.1 Problem Formulation
We consider the evolving domain generalization (EDG) in the domain-incremennal setting. For- mally, during training phase, we are given T sequentially arriving source (training) domains:
S S= s DD (D',D>, ....DD....,,D. D=), ). which are collected at timestamps 4 < l2 < .... < by, respectively. Each domain is defined as Dd = { y J. / = I,..., T, where æt is the i-th sample from the 1-th domain, v (0, 1, .,,.., C 13 is the category label of sample æ/. and No, C are the number of training samples in the (-th domain and the number of categories, respectively. In the domain- incremental setting, we can only access current domain Dt at timestamp & and historical domains (D',..., } are unavailable. This takes into account the data storage burden, privacy concerns and the dynamic evolution of the source domain. Following previous EDG works 124 29 2 the label set is the same among domains, but the data distribution of domains is assumed to continuously evolve over time in some patterns. And our goal is to generalize the model, which is composed of feature encoder Exp parameterized with ab and a classifier H parameterized with W, on unseen K DT+ }. To tackle this problem, we propose to model the evolving pattern at the parameter level via the conditional diffusion model and generate customized parameters for future domains by controlling the condition of the diffusion model.
target (testing) domains in the future: T = T
3.2 Diffusion Model
Diffusion models have achieved tremendous success computer vision by modeling the probability transformation from prior Gaussian distribution to the target distribution 14. 40). They typically
3

---

comprise a diffusion process to progressively add Gaussian noise to data in a multi-step Markov chain and denoising process to recover data from the noise via reversing the diffusion process.
Diffusion process. Given a clean data point 2o sampled from a real data distribution g(æ),, i.e., 20 (()), the diffusion process is characterized
Markov chain which slowly adds random Gaussian noise to zo in S steps, obtaining sequence of noisy samples; 2,,..., Ds. Formally, this process is expressed
(111s)/00) II (la,,aaa-- g(0r_(0,-1) -N /1 -1 1,
where {,8, E (0, 1)]==1 is a variance schedule, N represents Gaussian distribution, and is the identity matrix. And the forward diffused sample step denoted 2z. can be directly obtained in a single step by Eq. (2) without iteratively adding noise:
32- = vv + VI- E~N(O,
(2)
where a, = II B.). When step size approaches infinity, æs is equivalent to a data point from an isotropic Gaussian distribution, i..., the prior Gaussian distribution (O,
Denoising process. Given start noise 2 ~ (O. I), the denoising process moves backward on the multi-step Markov chain as s decreases from S to 1 to remove the noise at each step s, finally recovering the clean data. Concretely, the formulation of the denoising process at step is denoted as
B. Eo(x...) + OGE, VI ās E
Fz-1 Mg + O,E= V1
(3)
where (2 ().,)) V1-3, 0. VI-a, Eo (x.. 8) and (. is denoising model parameterized
with 0 to estimate the noise. 0g is variance hyperparameter that theoretically set to d² = 3, in most diffusion works 14, 26). During the training stage, the denoising model &g is trained by minimizing the following loss Laiss to minimize the noise estimation error:
Laiss E.... [|le Eelva.a0 + vI a,,,
(4)
Conditional diffusion model. The way of conditional diffusion models to generate samples is analogous to the unconditional one, except for the added conditional information. Specifically, as in most conditional diffusion works (25 17]. the denoising model & Eo 8) is replaced with & (ms, 8, c). where denotes the condition, e.g., class labels, texts, images, etc. The matched condition regulates the sample generation in supervised manner ensure the desired image content. And inspired by the generating of specific image contents via introducing conditional information to diffusion models, we propose to achieve domain-oriented parameter generation by controlling the diffusion condition.
4 Methodology
With the preliminary knowledge of EDG in the domain- incrrmmentrrll setting and diffusion models, we will present the details of Weight Diffusion (W-Diff) in this section. We begin with how to obtain the and then model the evolving pattern of parameters via the conditional diffusion model in Section 4.2 Finally, inference procedure W-Diff to generate
data for diffusion model training in Section 4.1
customized classifiers is presented in Section 4.3 The overview of W-Diff is illustrated in Fig.
4.1 Per-domain Parameter Fitting in Domain-Incremental Setting
In our approach, we try capture the evolving pattern optimized model parameters across domains and further generate customized parameters for target domain via leveraging the learned pattern. Considering the unaffordable training cost if modeling the whole parameters for relatively large models, we choose to excavate the evolutionary pattern in the task-specific head, e.g., the classifier for classification tasks. Nevertheless, the remaining parts of the task model would overfit to current domain and cause degraded generalization if without any processing. To avoid this, we learn domain-shared feature encoder for all domains and domain-specific classifier for each domain during the Comain-incrementi training. As t increases from 1 to I,, once the training stage

---

weight matrix current domain
weight matrix of historical domain
residual weight matrix
prototype matrix
slop gradient
concatenate
element-wise subtraction
D
reference point queue Qr
Weight Diffusion
(global)
(Training)
Training Domain
kpr-10
...
Pipre
D
noise
WWm
Task Model Training
Denoising U-Net E.
D
Q
Q
KV
KV
D
Feature Encoder
dilf
Ex
condition Sm
W
Testing Domain
Coon
fi
anchor point queue Qu (local)
WS
W
Wh h
o0O
T-2
jin
pe
prototype queus O, (local)
Figure 1: Overview of W-Diff. The reference point queue Q, stores classifier weights of recent |Qr] historical domains, and the anchor point queue Qa and prototype queue Qp store the updated classifier weights and prototype matrix at each iteration after the warm-up stage on current domain. f.on the prediction consistency loss to learn a domain-shared feature space and L is the cross-entropy loss. The conditional diffusion model Eg is trained with the noise estimation error loss Chiyy to model the evolving pattern of classifiers, conditioned on historical reference point and current prototype matrix.
domain D' ends, the classifier weights with the best validation performance on the validation set of D', denoted W E RRdx, is stored in the reference point queue Qr, where d, the dimension of deep features output by the feature encoder Feg. Qr global FIFO queue with the maximum length L. and is used to calculate the change of classifier weights between current domain and a given historical domain in Section 4.2 which reflects the evolving pattern at the parameter level.
Learning Domain-Shared Feature Encoder. In the domain-incremmnnall setting, we can only access the data from current domain, which prohibits from utilizing conventional DG methods that require to access multiple domains simultancously to learn invarininvariant feature representations. To tackle this problem, we resort to the stored different classifiers Qr- Intuitively, if domain-ssarre feature representation is learned, classifiers from different domains should give similar predictions for a given data sample. Hence, at timestamp f,, we train the task model on the 1-th domain D' by minimizing the consistency loss f to learn a domain-shared feature space:
1 N "C
KL(P!IPp Σ
N'
C.
1+ 121l
softmax(sg W x f!), t
E 2-=--e.
p 1+ 121l
P
(5)
softmax(sg( ( f!), I' t't'
where f Fap (ar)) R is the length the object and sg(-) denotes stopping gradients. W is the stored classifier weights of historical domain D and W is current classifier weights on D4.
Learning Domain-Specific Classifier. As increases from T, domain-specitf classifier is directly learned by incrementally training the task model via the supervision loss L on domain Dí:
1 N
C.e
CrossEntropy (softmax W f(), v1)
(6)
1=1
Overall, when training on domain D', the task model is optimized by the following total loss Li Cloual L + Act
(7)
where À is tradeoff hyperparameter to balance the two losses.
5

---

Collecting Data for Diffusion Model Training, Considering that parameters at the early stage are unstable, we start to collect the training data for diffusion model after the warm-up stage of the task model is over on corresponding domain. Specifically, when training on the I-th domain, the warm-up epochs of the task model account for p E (0, 1) of the total training epochs on domain Dr, where o is a hyperparametter After the warm-up stage, the updated classifier weights W E RCxd, via back- pook-propggating the gradients of Liotal and the updated prototype matrix je' E RCxd via Eq. (8) are respectively stored into the anchor point queue Qa and prototype queue Qp at each iteration.
wllcl n² 10
+ EE Id f , 0, 0,
n¹ +
n' = n' + B.
(8)
u"|c] is the c-th row of p², i.c., the prototype of class c on domain D' based on all data in the seen batches after the warm-up stage on Dt B is the batch size of task model. n' counts the total number of samples in seen batches after the warm-up stage and initialized to at the start of training on each domain. p." [c] is the predicted probability of a belonging to class e via current classifier of domain D Here, using predicted probability instead of ground-truth label to compute prototypes avoids the issue of missing categories in a batch and the inaccessibility of labels in testing domains.
In implementation, Qa and Qp are both FIFO queue with the maximum length M, i.e., the training batch size of the diffusion model. When they are full, the training of the conditional diffusion model starts. Note that Qa and Qp are only used during the training phase and are locally used on each domain. That is, they are initialized to empty the beginning of the training stage on each domain.
4.2 Modeling Parameter Evolution Pattern with Conditional Diffusson Model
Having prepared the data for diffusion model training. we can utilize them to learn the evolving pattern of parameters during the Nomain-incremente training process of the task model. To be specific, we use the difference between the classifier weights of current domain and that of a given historical domain to represent the evolution of parameters:
AW m = W W
where W m. 171 = 1,2....,MM. M, is the classifier weights of current domain Dt, which is cached in the anchor point queue Qa when training the task model on Dt. And W is the classifier weights of the historical domain D" from the reference point queue Qr. t' E {t - IQrl...., t 1}. Hence, the residual classifier weights Wt,'', Jm=1 represent how to evolve from reference point to the anchor point. Moreover, to guide the evolution, we provide the paired condition for each residual classifier weight matrix Wt.!' m where the paired condition formulated as c W $ yr E Rxxd, x2 and @ denotes concatenating. The additional condition provides the information about the the starting point and knowledge about the distribution of data in the feature space. which is rightly the anchor point, i.c., the optimized classifier, needs to adapt to. Then, when the task model is incrementally trained on the i-th domain, the conditional diffusion model is also incrementally trained to minimize the following noise estimation error loss Cåisz f in Eq. (10), as learn how to generate the desired residual classifier weights when given the reference point and prototype matrix as the condition.
Cafs = E Ew' W..... EQ. EQ..ee [||e - EolVo,, AWAY' m + + ch {1p] (10)
In implementation. the conditional diffusion model adopts the similar U-Net structure as LDM [31 and uses hybrid conditioning way, i.e., the condition is injected both in the cross-attention and input sides. Eq. (TO). different reference points could enrich the diversity of training data for the conditional diffusion model and provide the evolving pattern different time intervals.
4.3 Generating Customized Classifiers in Inference Phase
After finishing the training on domain DT, we can use the conditional diffusion model to gen- erate customized classifiers for given testing domain Diest, Firstly, we calculate the proto-
Noter plest[c] fleet, where plest type matrix eest of domain Diest via 4 test [c] = nlor Eiet D
TQ. Eweed EQ. softmax WT x fleat), 0, O, C 1. Here, we use the more robust average

---

prediction of multiple classifiers to compute the prototype matrix in the inference phase. Then, given each reference point Wt in Qr along with the prototype matrix f p we can generate Mg residual 1
classifier weights: (A Jj=1 by substituting the denoising net in Eq. (3) with its conditional version and applying the denoising process with condition glest, W jutest, Ultimately, use the following average weight ensemble Wiest as the final classifier for label predicting on Diest
Wetstt
1 11X [21 Mg Ewwre E
My "W"" AW Ex
(11)
In this way, capitalize the powerful modeling and generating ability of conditional diffusion model to cheaply produce a great number of target-customizer classifiers, which offers more robust and accurate predictions. The pseudo codes of training and testing procedures are in Appendix C
5
Experiments
5.1 Experimental Setup
Benchmark Datasets. We evaluate W-Diff on both synthetic and real-world datasets includ- ing two text classification datasets (Huffpost, Arxiv), three image classification datasets (Yearbook, RMNIST, (MoW) and two multivariate classification datasets (2-Moons, ONP). Except for syn- thetic datasets 2-Moons and RMNIST that the rotation angle proxy for time, all other datasets collect real-world data with the distribution shift over time. Following [45] the number of source and target domains is set Yearbook: (T 16, K 5), RMNIST: (T
K 3), fMoW:
(T 13, K = 3), Huffpost: (T = 4, K = 3). Arxiv: (T = 9, K = 7). 2-Moons: (T = 9, K = 1), ONP: (T 5, K 1). For each source domain,
randomly divide the data into training and validation sets in the ratio of 9 1. For more details on datasets, please refer to Appendix [.I]
For the conditional diffusion model, we implement it in a U-Net similar to LDM 31. Please refer to Appendix D.2 for details.
Network Details. For the task model, we follow the usage in 48 45
Training Details. For all datasets, we set the batch size B = 64, the loss tradeoff À 10 and the maximum length L for the reference point queue Q,- To optimize the task model, the Adam optimizer with momentum 0.9. As for the warm-up hyperparameter p. p 0.6 for Huffpost, fMoW and p = 0.2 for Arxiv, Yearbook, RMNIST, 2-Moons, ONP. For the conditional diffusion model, we set the maximum diffusion step 5 = 1000 and use the Adam optimizer with batch size M = 32, where M also maximum length of queue Qa and Op. And the number of generated residual classifier weights based each reference point is set to M, 32 in the inference stage. All experiments are conducted using the PyTorch packages and run on a single NVIDIA GeForce RTX 4090 GPU with 24GB memory. Three independent experiments with different random seeds are repeated for each task to report the mean and standard deviation (std) of accuracy, which is denoted in the format of "mean + std" in the table. Please refer to Appendix D.3 for more details.
adopt
Table 1: Accuracy (%) Huffpost and Arxiv. The best and second best results in the incremental setup are bolded and underlined, respectively, (Huffpost: K = 3, Axriv: K 7)
Huffpost Accuracy (%)
ADXIV Accuracy (%6
Incremental training
Access multiple domains
Method
D
0OD
DT+1
woest
avg.
worst
IRM 1
71.04
70.31
68.97
51.11
45.89
42.86
CORAL
71.34
70.08
68.68
50.98
45.77
42.71
Mixup 35
73.34
71.16
69.29
57.58
$2.77
49.62
A.ISA
49
72.19
70.24
68.60
56.53
$2.41
49.67
Inchinctune
X
23.57
71.98
69.30
36.22
$2.43
49.37
Mixup 55
73.07
56.64
52.95
FWC
73.64
71.53
68.99
56.60
$2.78
49.73
SI [53
72.58
71.50
69.61
49.98
47.27
44,77
72.23
DRAIN
73.42
71.75
69.69
$6.04
$2.07
48.97
Evoš
73.42
72.36
56.60
W.Diil
73.1
72.29
70.40
36.66:10.11
S3.311010
50.70
10.20

---

Table 2: Accuracy (%) Yearbook, RMNIST and fMoW. The best and second best results in the incremental setup are bolded and underlined. (Yearbook: 5, RMNIST: 3, fMoW: K 3)
BMal
Incremental tnining
mallipl:
Access
29742
131-1
302.
95.10
64.77
CORAL
Maxup
35.52
CIDA
38.45
97.
GI
97.7%
61,62
LSSAE 29
88:75
96.73
59.15
45.25
Mixap [SS
90.21
SuaCL&
93.94
39.65
64.97
SwAY
91.46
98.05
BWC 16
93.64
98.56
66.23
54.55
SI [3]
97.00
04.67
93.45
98.61
93.27
85.65
66.61
$4.59
46.46
A-GEM [
94.36
90.96
55.65
95.99
86.95
75.45
54,54
47.61
41.15
SGP [+]
95.63
92.92
91.39
07.12
85.97
75.00
DRAIN M
96.23
94.71
95.75
98.52
93.09
83.75
67.22
$5.05
46.24
Evol [5
97.37
95.53
94.78
98.64
03.84
$7.04
67.38
$4.44
45.86
W-Ihit
9505=0.17
98.7
64.2290013
87.36
68.00
46:51:00355
Table 3: (a): Error rate (%) on 2-Moons and ONP (K = 1). (b): Ablation study on RMNIST.
Method
Offine
22.4:446
M.Mooms
Erroe rate ($)
33.8
ONP
Method
Conditioning way For Eg
c.om
Loss
Q.
Computed Wiese based on
noise adiedd
Q+
diffusion
DT+1
Accuracy (%)
worst
LasDomaii
14,910,9
36.020.2
variant A
hybrid
98.48
91.69
81.87
16.7133.4
98.55
CDOT 27
CIDA
10.8::1.6
9.341.0
34.120.0
34.7:00.6
variant C
98.47
93.82
87.28
24
3.4+14
36.4=0.8
variant
98.55
93.78
87.13
DRAIN
3,2++12
38.3+1122
cross, altn
97.98
83.19
Evo8 [45]
2,5+1..0
variant
concal
98.44
92.93
85.03
W-Diif
13++10
32.9-0.5
W-Diff
hybrid
98.70
94.12
87.36
(a)
(b)
Evaluation Metrics. We report the generalization performance K target domains in the future, including the average accuracy "OOD avg." ( Σk Accuracy(D"*** and the worst accuracy "OOD worst" (minke!l,.... Accuracy(DD on K target domains and accuracy on DT+1,
5.2 Main Results
We provide the quantitative results in Table 1 2 3(a),
a), where the results of baselines the non- incremental and incremental scenarios are reported from 45 For ONP dataset, we notice that previous continuous domain adaptation methods (CDOT and CIDA) and EDG methods (GI and DRAIN) all perform worse than the Offline method that trains the task model on the cumulation of all source domains. In contrast, our W-Diff still obtains superior accuracy to previous state-of-the-at method (EvoS) this challenging dataset, which validates the superiority of W-Diff. Besides, W-Diff also achieves the best results on Huffpost, Arxiv, R.NIIST and fMoW, in terms the OOD worst accuracy. These results benefit from the modeling of parameter evolution pattern and the more robust predictions via the weight ensemble based on the conditional diffusion model. For Yearbook dataset, DRAIN, which models the evolution of whole model parameters via LSTM, is inferior to EvoS, which models the evolution of domain-level feature distribution. It suggests that modeling the evolving pattern at the feature level may be more appropriate for Yearbook, which also explains why W-Diff does not obtain the state-of-the-art performance on Yearbook. But our W-Diff still obviously outperforms DRAIN. Overall, we can observe that W-Diff' surpasses the baselines in the seremental-trainiing setup on six out of seven datasets, which shows the superiority of W-Diff.
5.3 Analytical Experiments
Ablation Study. Firstly, the significant performance drop of variant A in Table ((b) suggests that learning domain-invariant feature representations necessary for EDG in the setting. Otherwise, the feature encoder could easily overfit to current domain, prohibiting the task model from generalization. Then, we try different ways to construct the average weight ensemble Wiest including variant C which directly uses the historical classifier weights in Qr. i.e., Wiestt
[..1 Ewreq, Wt' and variant D which augments classifier weights by adding small

---

D'y=1
y==
W-Diff
W-Diff
Evo5
Evo5
t=
t=
RMNIST
Visualization decision boundary on 2-Moons.
(b) t-SNE visualization of features.
Figure 2: (a): Decision boundary of EvoS 45 and W-Diff on 2-Moons, where we incrementally train the model until the 1-th domain and then visualize the decision boundary for future domain Dr+11 (b): visualization of features from target domains, where different colors represent different domains.
W
W
WIP
W
W!
W?
W
W7
WP5
W
W
W'
W
( 1 Exd
0 WiGHT
OCD
OGD W-DI Offiwis
0O0 wont: w-alf
DOO wont: Exe5
0C0
000 worst Dffiownn
99.0
94.5
60.0
Accuracy 196
98.75
98.5
72.5
57.5
70.0
2 40.0
72.0
87.0
97.5
71.5
86.5
97.3
710
58.0
98.60
96.5
RMNIST
98.55
WT W75
95.0
Huffocst
Huffpos::
Huffpost
(a) Visualization of classifier weights.
(b) Results when evaluating in batch-data stream.
Figure 3: (a): Visualization of classifier weights for DT+1 +1 T = 6, on RMNIST and their accuracy range. W t' = 1....,6,, 6, is the reference point from Qr, V/7''' is the generated Mg classifier weights based Wi' and W is the average weights of 'D7 fine-tuned classifier weights in the last 200 iterations. (b): Accuracy of EvoS and W-Diff on RMNIST and Huffpost datastes, where W-Diffsatream denotes W-Diff evaluated with batch-data stream for each target domain.
noises to the weights in Qr, i.e., Wtest TQ, Mr Ew'eq, EØe (W"' +noise;), noise, Uniform(00.0.. 0.01). The inferior results of variant B, C and D indicate that W-Diff benefits from generating meaningful and customized classifier weights via controlling the condition of diffusion model. Finally, different conditioning ways for the diffusion model are explored, including variant E which injects the condition only in the -ross-attention, and variant F which injects the condition only in the input by concatenating the condition with diffused residual classifier weights. Results of variant E and are both unsatisfactory. This probably due the large gap between the residual classifier weights in the input side and the full classifier weights in the condition side, which makes the information interaction hard. And injecting the condition only on the input side can lead to insufficient information interaction. Empirically, we find that the hybrid manner works best.
Decision Boundary Visualization Future Domain. In Fig. 2(a) the model is incrementally trained until the training stage on the 1-th domain D' finishes. Then we visualize the decision boundary the next future domain Di+1, 8, From the results, we can see that the decision boundary of W-Diff adapts to the evolution domains better than that of EvoS, which shows the superiority of W-Diff' addressing evolving domain generalization in the domain-ineremenn
setup. I-SNE Visualization of Features. In this qualitative experiment, we visualize the features future target domains for RMNIST dataset to show the effectiveness of Lt From Fig. 2bb we can observe that features from different target domains align well. To some extent, it verifies the effectiveness of C. to learn domain-shamme feature space, which contributes to the mitigation of distribution shift.
Visualization Generated Classifier Weights. In Fig. 3(a) we plot the generated classifier weights for domain D 6, on RMNIST, well as the accuracy range on DI+1 of generated classifier weights based on different reference points. In Fig. [[a] some generated weights locate close to
the average fine-tuned weights W showing that W-Diff generates domain-customize. classifiers.

---

Table 4:
Accuracy (%) of W-Diff' RMNIST dataset using different conditions. (K' =3)
Accuracy (%)
Method
Condition
D
OOD avg.
OOD worsl
W-Diff
reference point + prototype matrix
98.70
94. 12
87.36
W-Diff
scaled reference point + prototype matrix
98.69
94.17
87.46
t denotes the reference point is scaled by the factor 1) 1.5 difference between the reference point and anchor point.
146-AT where At the timestamp
Besides, W W7/2, W are similar, while W W715 W7/6 are different, due the more pronounced differences among reference points W W and W This implies that the evolution pattern may be different at different time intervals. And these diverse and tegh-performing generated weights based on different reference points could conduce to more robust predictions.
Evaluating in Batch-Data Stream. In addition to the inference way in Section 4.3 also provide another version, where the data in target domain arrives batch by batch. Concretely, we use the iterative manner in Eq. along with the average prediction in Section 4.3 to update the prototype matrix test once a batch of data from Diest arrives. Then, we compute the average weight ensemble Wtest via Eq. for this data batch. Fig. 3(b) shows the results on RMNIST and Huffpost. The two manners present similar results and users can choose appropriate manner based on their scenarios.
Equipping Condition with Timestamp Difference. In this part, try to explicitly incorporate the timestamp difference between the anchor point and reference point into the condition of diffusion model. Concretely, we scale the reference point W in the condition c'.t' by a factor 7, where " 1.5 1+e-Ae and At = t by is the timestamp difference between reference and anchor points. More distant reference points have larger At and weakened. The results RMNIST dataset are given in Table 4 where the average and worst accuracies of target domains improve slightly. The insignificant performance improvement may be due the fact that approach implicitly takes the timestamp difference into account via the domain-incremental training and residual classifier weights.
Results with Larger Backbones. We try larger backbones on the foo dataset by replacing the DenseNet-121 with DenseNet- 169/201/161 [1S] respectively. The results are provided Table 5 When applying lagers backbones, our method still works well and further performance improvements are obtained. This benefits from the consideration of only modeling the evolution of classifier weights, instead of the whole network parameters. Otherwise, the training difficulty and huge memory burden from the conditional diffusion
Table 5: Accuracy (%) of W-Diff on fMoW dataset with differ-
ent backhones. (K=33 =3)
Accuracy (%)
Backbones
parameters
DT
OOD
O0D
avg.
DenseNet-121 (growth rale=32)
64.5MB
68.80
55.86
46.5T
DenseNet-169 (growth rale=32)
114.4MB
70.20
56.81
47.50
DenseNet-200 (growth rate=32)
161.8BB
70.38
56.28
46.40
DenseNet-161 (growth ratc=48)
230.8MB
71.28
57.36
47.33
model would be unbearable, despite of the greater feature extraction capability of larger backbones.
Conclusion
This work delves into the under-explored problem of evolving domain generalization in the domain- incremental setting, where the source domain is also non-stationary and dynamically evolves. To tackle this, we propose a Weight Diffusion (W-Diff) approach to capture the evolving pattern across domains at the parameter level and further generate customized classifiers for future domains. W-Diff innovatively leverages the conditional diffusion model to learn the evolution of classifiers from historical domain to current domain, conditioned the historical classifier weights and current prototype matrix. Extensive results on synthetic and real-world datasets verify the efficacy of W-Diff.
Acknowledgements
This paper was supported by National Key R&D Program of China (No. 2021 YFB3301503), the National Natural Science Foundation of China (No. 62376026), and also sponsored by Beijing Nova Program (No. 20230484296), CCF-Tencent Rhino-Bird Open Research Fund and KuaiShou.
10

---

References
[1] M. Arjovsky, Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv:1907.02893 2019.
[2] G. Bai, C. Ling, and L. Zhao. Temporal domain generalization with drift-aware dynamic neural networks. ICLR, 2023.
[3] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
[4] H. Cha, J. Lee, and J. Shin. Co²1: Contrastive continual learning. In ICCV, pages 9496-9505. 2021.
[5] A. Chaudhry, M. Ranzato, M. Rohrbach, and M. Elhoseiny. Efficient lifelong learning with A-GEM. In ICLR, 2019.
[6] Chen, S. Kornblith, M. Norouzi, and G. E. Hinton. A simple framework for contrastive learning of visual representations. In ICML, pages 1597-1607, 2020.
[7] L. Deng. The MNIST datahase of handwritten digit images for machine learning research [best of the web]. SPM, 29(6):141-142 2012.
[8] Y. Du. Z. Xiao, S. Liao, and C. Snock. Protodiff: Learning to learn prototypical networks by task-guided diffusion. In NeurIPS, 2023.
[9] Ebrahimi, E. Meier, R. Calandra, Darrell, and M. Rohrbach. Adversarial continual learning. In ECCV, volume 12356, pages 386-402, 2020.
[10] E. Fini, V. G. T. da Costa, X. Alameda-Pinede E. Ricci, K. Alahari, and J. Mairal. Self- supervised models are continual learners. In CVPR, pages 9611 9611-9620, 2022.
[11] R. Gao and W. Liu, DDGR: continual learning with deep liffusion-based generative replay. In ICML, pages 10744-10763, 2023.
[12] H. Grossniklaus, M. Nickerson, alterations in aging and age-related discases of the eye. IOVS, 34L14))ORSF2--RSFF22 2013,
Edelhauser, L. A. Bergman, and L. Berglin. Anatomic
[13] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[14] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[15] G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In CVPR, pages 2261-2269. 2017.
[16] Kirkpatrick, R. Pascanu, N. C. Rabinowitz. J. Veness, G. Desjardins, Rusu, K. Milan, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell. Overcoming catastrophic forgetting in neural networks. arXiv:1612.00796 2016.
J. Quan, T. Ramalho, A.
[17] G. Kwon and C. Ye. Diffusion-bassee image translation using disentangled style and content representation. In ICLR, 2023.
[18] J. Lee, Y. Lee, J. Kim, A. R. Kosiorek, S. Choi, and Y. W. Teh. Set transformer: A framework for attention-basse permutation-invarian neural networks. ICML, volume 97, pages 3744-3753, 2019.
[19] H. Li, S. J. Pan, S. Wang, and A. C. Kot. Domain generalization with adversarial feature learning. In CVPR, pages 5400-5409, 2018.
[20] S. Li. C. H. Liu, Q. Lin, Q. Wen, L. Su, G. Huang, and Z. Ding, Deep residual correction network for partial domain adaptation. TPAMI, 43(7):2329 2344, 2021.
[21] S. Lutati and Wolf. Ocd: Learning to overfit with conditional diffusion models. In ICMI., pages 23157-23169 2023.

---

[22] F. Lv. J. Liang, S. J. Zhang. and D. Liu. Improving generalization with domain convex game. 2023.
In CVPR. pages
[23] K. Muandet, D. Balduzzi, and B. Schölkopf. Domain generalization via invariant feature representation. In ICML, pages 10-18, 2013.
[24] A. Nasery, S. Thakur, V. Piratla, A. De, and S. Sarawagi. Training for the future: A simple gradient interpolation loss to generalize along time. In NeurlPS, pages 9918--19099 2021.
[25] H. Ni, C. Shi, K. Li, S. X. Huang, and M. R. Min. Conditional image-to-vides generation with latent flow diffusion models. In CVPR, pages 18444-18455, 2023.
[26] A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In ICML, pages 8162-8171, 2021.
[27] G. Ortiz-Jiménez M. E. Gheche, E. Simou, H. P. Maretic, and P. Frossard. CDOT: continuous domain adaptation using optimal transport. arXiv: 11448, 2019.
[28] W. S. Peebles, I. Radosavovic, T. Brooks, A. A. Efros, and J. Malik. Learning to learn with generative models of neural network checkpoints. arXiv:2209.12899 2022.
[29] Qin, S. Wang, and Li. Generalizing to evolving domains with latent structure-aware sequential autoencoder. In ICML, pages 18062-18082, 2022.
[30] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, G. Krueger, and Sutskever. Learning transferable visual models from natural language supervision. In ICML, volume 139, pages 7748-8763,, 2021.
[31] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10674-10685, 2022.
[32] G. Saha and K. Roy. Continual learning with scaled gradient projection. In AAAI, pages 9677-9685, 2023.
[33] V. Sanh, L. Debut, J. Chaumond, and Wolf. Distilbert, distilled version of BERT: smaller, faster, cheaper and lighter. 1rXiv: 108, 2019.
[34] Serrà, D. Suris, M. Miron, and A. Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In ICML, volume 80, pages 4555 2018
[35] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[36] B. Soro, B. Andreis, H. Lee, S. Chong, F. Hutter, and S. J. Hwang. Diffusion-based neural network weights generation. arXiv:2402.18153, 2024.
[37] B. Sun and Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV Workshops, pages 443 450, 2016.
[38] R. Taori, A. Dave, V. Shankar, N. Carlini, B. Recht, and I.. Schmidt. Measuring robustness natural distribution shifts in image classification. In NeurIPS, 2020.
[39] R. Tiwari, K. Killamsetty, R. K. Iyer, and P. Shenoy. GCR: gradient coreset based replay buffer selection for continual learning. In CVPR, pages 99-108, 2022.
[40] A. Ulhaq, N. Akhtar, and G. Pogrebna. Efficient diffusion models for vision: A survey. arXfv.2210009292, 2022.
[41] H. Wang, H. He, and D. Katabi. Continuously indexed domain adaptation. In ICML, volume 119, pages 9898-9907, 2020.
[42] K. Wang. Z. Xu, Y. Zhou, Z. Zang, T. Darrell, Z. Liu, and Y. You. Neural network diffusion. 04Xiv:2402.133 13144, 2024.
[43] Y. Wang, 7. Huang, and X. Hong. S-prompts learning with pre-trained transformers: occam's 's razor for domain incremental learning In NeurIPS, 2022.
12

---

[44] B. Xie, Y. Chen, J. Wang, K. Zhou, B. Han, W. Meng. and J. Cheng. Enhancing evolving domain generalization through dynamic latent representations. In AAAL, pages 16040-16048, 2024.
[45] M. Xie, S. Li, L. Yuan, C. H. Liu, and Z. Dai. Evolving standardization for continual domain generalization over temporal drift. In NeurIPS, 2023.
[46] L. Yang, Y. Han, X. Chen, S. Song. J. Dai, and G. Huang. Resolution adaptive networks for efficient inference. In CVPR, pages 2366-2366 2375,2375, 2020.
[47] L. Yang. H. Jiang. R. Cai, Y. Wang. S. Song, G. Huang, and Q. Tian. Condensenet V2: sparse feature reactivation for deep networks. In CVPR, pages 3569-3578, 2021.
[48] H. Yao, C. Choi, B. Cao. Y. Lee, P. W. Koh, and C. Finn. Wild-time: A benchmark of in-the-wiit wild distribution shift over time. In NeurlPS, 2022.
[49] H. Yao, Y. Wang, S. Li, L. Zhang, W. Liang, Zou, and C. Finn. Improving out-of-distribution robustness via selective augmentation. In ICML, pages 25407-25433 2022.
[50] Q. Zeng, C. Shui, L.-K. Huang, P. Liu, X. Chen, C. Ling, and B. Wang. Latent trajectory learning for limited timestamps under distribution shift over time. In ICLR, 2023.
[51] Q. Zeng, W. Wang, F. Zhou, C. Ling, and B. Wang. Foresee what you will learn: Data augmentation for domain generalization in non-stationary environments, arXiv:2301.07845, 2023.
[52] Q. Zeng, W. Wang, F. Zhou, G. Xu, R. Pu, C. Shui, C. Gagné, S. Yang. C. Ling, and B. Wang. Generalizing across temporal domains with koopman operators. AAAI, pages 16651-16633 2024.
[53] F. Zenke, B. Poole, and S. Ganguli. Continual learning through synaptic intelligence. In ICML, pages 3987-3995, 2017.
[54] B. Zhang, C. Luo, D. Yu, X. Li, H. Lin, Y. Ye, and B. Zhang. Metadiff: Meta-learning with conditional diffusion for few-shot learning. In AAAI, pages 16687-16695, 2024.
[55] H. Zhang, M. Cissé, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk mini- mization. In ICLR, 2018.
[56] K. Zhou, Z. Liu, Y. Qiao, T. Xiang, and C. C. Loy. Domain generalization: A survey. TPAMI, 45(4):4396-444 -415, 2023.
[57] K. Zhou, Y. Yang, Qiao, and T. Xiang. Domain generalization with mixstyle. In ICLR, 2021.
13

---

Appendix Contents
A. Broader Impacts & Limitations
B. Notation Table
C. Algorithm of W-Dill
D. Experimental Setup Details
E. More Results
A Broader Impacts Limitations
Broader Impacts. In this work, explore the evolving domain generalization in the domain incremental setting. The ability to continually learn from dynamic source domains and leverage the learned evolving pattern generalize unseen domains in the future may benefit relevant stationary scenarios, e.g., advertisement recommendation with continually emerging new training data and autonomous driving with distribution shift time geographical position, etc. It reduces the time and cost for labeling data of target domain and avoids the low efficiency of training the model from scratch with all saved domains once demanding scenarios, the prediction from the model should be adopted with caution to avoid severe accidents, as failures can occur in our method when facing significant distribution shifts.
training domain is available. Yet, for high-security
Limitations. Our work presents a way to capture the evolving pattern at the parameter level via capitalizing the powerful modeling ability of conditional diffusion model. Yet, like any research, our work is not absolutely perfect. There are indeed some limitations that should be acknowledgen Firstly, the task considered in this paper limits the classification. In the future, we may extend method to more diverse tasks, e.g., regression tasks. Besides, considering the training cost, we only model the evolution of classifiers. Perhaps, it feasible to consider parameters by mapping them into low-dimensional latent space, but achieving the accurate encoding and decoding is not easy. We leave this for future work.
B Notation Table
Given the large number of notations used throughout the paper, we provide an overall notation description in Table to ease the burden on readers.
Table Notation description
Data-related
T
the number of source domains the number target domains the number of categories
C
the index of categories (c E 10. 1)) the index of domains (t, t' (1,2,.... KK) the /-th domain
t
D
the timestamp which the t-th domain is collected at the number of samples in the t-th domain
N
the i-th sample in the f-th domain
the category label of the i-th sample in the t-th domain (y² E 10.1,..,,C-
S
sequence of source (training) domains (S (D), D2 . Dd
T
a sequence of target (testing) domains (T = (DT+1 (TT, testing domain (Dtest E T)
DTK
Diest
Task model-related
batch size of task model feature extractor
Exp
zb
parameters of the feature extractor
d
the dimension of of deep features output by the feature encoder
f
the deep features of the i-th sample in the t-th domain (f = E (x!) E Rdd)) classifier
fw
14

---

W
parameters of the classifier the reference point queue the maximum length of Qr the saved classifier weights in Q, for the l'-th domain (W E RRxdy) the current classifier weights of the t-th domain (W E Rx)d)) the current prototype matrix of the t-th domain (μ² IR (xxd) the c-th row μ
Q
L
We'
W
u'
u'ld
the total number of samples in seen batches after the warm-up stage domain D
the prediction for the i-th sample in the t-th domain by the classifier of domain D
pa [c]
the e-th element of p; f.t"
d
the average prediction for the i-th sample in the t-th domain the anchor point queue, which stores the classifier weights of current domain
Qa
Qp
the prototype queue, which stores the prototype matrices of current domain the maximum length of Qa and Qp the index of the object in Qa and 2, (m E 21, MM)
M
the m-th prototype matrix in the prototype queue 2p of the t-th domain
W
the m-th classifier weights in the anchor point queue Qa of the 1-th domain residual classifier weights (A W Wt Wt'
A AWLe'
condition of the conditional diffusion model (c,,." W pm RCx)(2) warm-upp hyperparameter (p E (0, 1))
tradeoff hyperparameter
C
the consistency loss the t-th domain the cross-entropy loss on the t-th domain
total
the total loss the I-th domain (Cigtal L.e X. Diffusion model-related
diffusion steps
s
the index of diffusion steps (s, s' E (1, distribution of variable æ
(xx
Co
the original data point from g(x)
a.
the noisy data point at the s-th diffusion step
B,
the variance used at the s-th diffusion step (3, (0, 1))
the product of all (1 3yr) until the s-th diffusion step (ā, = - (1 Br())
random noise denoising model parameters of the denoising model variance hyperparameter condition of the denoising model noise estimation error loss on the t-th domain
Eg
Caies
Mg
the number of generated classifier weights based on each reference point the estimated prototype matrix the testing domain Diest during inference stage the condition of diffusion model for domain Diest (cfest, = W + yeest)
p eest
test,
A Whest
the j-th generated residual classifier weights, conditioned on clest.t' the average ensemble weights for the testing domain Diest Others
identity matrix
N(...)
Gaussian distribution
mathematic expectation real number 1.2-Norm
R
the length of an object
KL(-I))
Kullback-Leible divergence
sg(.)
stopping gradients of an object
softmax(()
normalized exponential function
concatenating operation
i.j. l
indices
15

---

C
Algorithm of W-Diff
The training and testing procedures of W-Diff are presented in Algorithm [and] 2
Algorithm 1: Training procedure for W-Diff
Input:
feature encoder Enp- classifier l/w. conditional diffusion model Es. reference point queue Q, with length L, anchor point queue Qa with length M, prototype queue Qp with length M, batch size B, loss tradeoff hyperparameter X, warm-up hyperparameter p, maximum diffusion step S, training iterations IrS of task model, inner iterations I JDM for updating diffusion model. Initialize model parameters v/ as 26", W as W 0 as f and set Q, =
sequentially arriving source domains = (DI D2 D D"),
for f to do
Set 40* 2/² W W 0' 0 ..Qa 0, Qp = 0, n = 0, n' O.
4
for iler - to ITS do
Ciooaal
Randomly sample batch of data (æi, ud _ from domain D
Get the deep features of samples: {fi = Ex (<)))
Calculate the supervision loss C in Eq.
Ciota! + C.ee
10
if > then
11
Calculate the consistency loss C.om in Eq. 5
12
fiotal+= L'
Update ub" and W by backpropagating the gradients of Liotal if (iter p. .rrs A (t 1) then
13
15
Update prototype matrix μe via Eq. 1
16
Push W into Qa and 24 into Qpi Qa t- W Qp + n²:
17
Qal - then for inner_iler to
Jpm do QQ.1
18
19
Sample diffusion step size Uniform(L...... S) and e (0,1). Calculate the noise estimation error loss Llass via Eq. (TO).
20
Update 0f by backpropagating the gradients of Cassf
21
22
Push the classifier weights with the best performance on the validation set D', denoted as Wt into Qri Q f- W
return Final T² 0" ,r,
23
D Experimental Setup Details
D.I Dataset Description
comprises 63, 907 news headlines from the Huffington Post, with the time span from 2012 to 2018. These news headlines belong to 11 categories: "Black Voices", "Business", "Comedy". "Crime", "Entertainment" "Impact", "Queer Voices". "Sci- ence", "Sports". "Tech" and "Travel". This dataset reflects changes in news content and style over time. Follwoing [45), the first 4 years used for training (T' = 4) and the last 3 years used for testing (K 3). For each training domain, we randomly divide the data into training set and validation set in the ratio of 9: 1
Huffpost (license: CCO: Public Domain) from 48
a large-scale dataset, including 2, 057, 952 paper titles from 2007 to 2022. It reflects the change over time as research fields evolve. The task is to classify a research paper into one 172 categories based solely on its title. For this dataset, we use data from the first 9 years as source domains (T 9) and data from the last 7 years target domains (K 7). For each source domain, the data is randomly divided into training set and validation set in the ratio of 9
Arxiv (license: CCO: Public Domain) in
16

---

Algorithm 2:
Testing procedure for W-Diff'
Input: sequentially arriving target domains T = (DT+1, DT+2 DT+K) feature encoder EgT, conditional diffusion model ,Ør, reference point queue Q,, number of categories C, batch size B, maximum diffusion step S, number of generated residual weights Mg based on each reference point.
for k == to K do
2 Set Diest = D*+
3 Calculate the prototype matrix plast vai peesto [c] = Nictt
1 EN P pinst(ci flcst where Dleat [Q-1 Ewe'e. softmax(We x ftoat), flest Eap (x) (2ffest), ) C for W EQ, do
Generate M, residual classifier weights: {AW) رررل by substituting the denoising
net in Eq. (3) with Egr and applying the denoising process with condition clest,t' w fest
Obtain the average weight Wiestt 12- Mr Ew'eq, EQ. Σ
j=1 + A A )
Get the final label predictions on domain Drest: fotest = argmax, pest [c]] Noxt
where
plest softmax Wrest flast).
8 return Label Predictions {1/ + N KK Ji=1
Yearbook (MIT license) dataset comes from 48 It collects 37, 189 grayscale yearbook photos from 128 American high schools, with the time span from 1930 to 2013. The resolution of photos 32 32. Photos from different years reflect changes in fashion trends and social norms over the decades. The task is to classify the genders from a yearbook photo. It is worth mentioning that we only use this dataset to evaluate the generalization performance of different methods in classification tasks. Following (45), we group the data into domains at four-year intervals, resulting in 21 domains. And the first 16 domains are used as source domains (T = 16). with the last domains as target domains (K = 5). For each source domain, we randomly select 90% of samples as the training split and 10% of samples the validation split. And for each target domain, we evaluate on its all data.
RMNIST (license: CC BY-SA 3.0) is constructed from MNIST dataset 7 which contains grayscale images of digits from 0 to 9. The image resolution is 28 28. RMNIST first randomly divides all data in MNIST into 9 groups and then creates 9 domains by rotating the 9 groups by 09, ................................ 80%, respectively. The rotation angle is used simulate the evolving data distribution over time. Following [4). we use the first 6 domains (T = 6) as source domains and the last 3 domains as target domains (K 3). Similarly, split each source domain into training and validation sets in the ratio of 9
dataset is from 48). which consists of 141, 696 RGB satellite images from 2002 to 2017. The visual features in these satellite images change over time due to human and environmental activities. The image resolution is 224 x 224 and the task is classify the functional purpose of the buildings land in image into one of 62 categories. For this dataset, we consider each year as a separate domain. And the first 13 domains are used for training (T 13), while the last domains are used for testing (K 3). The ratio of training data to validation data for each source domain is 9 1.
MMo (license: https //ti git.t...hh..... com fMoo/ dwwtaetttbalooooo wwt./////aaaeee/ee/ti.e...
2- 2-Moons (license: https: //gi:: gitbbtthh....... com/ Bai Tweaaeeeeettttte BacbeeBsst/RRAATI wwtt////w/////////n/eeaeeeeee
from 2 is constructed from 2-entangled moons dataset, where the lower moon with label O and the upper moon with label contain 100 data points, respectively. 2-Moons creates 10 domains by counter- clockwise rotating the 200 data points at an interval of 18 Similar to RMNIST, the rotation angle simulates the evolving of data distribution. Following (2, the first 9 (T 9) domains are used as source domain and the last domain is used as target domain (K =1)1
Online News Popularity (ONP) (license: CC BY 4.0) in [2] summarizes heterogeneous set of features related to the articles published by Mashable in a two-year period. This dataset is divided into 6 domains by time, and the goal is to predict whether an article is popular in social networks based on its features. Following 22) we use the first domains for training (T = 5) and the last domain for testing (K- = 1).
17

---

Table 7: Configuration of the U-Net Eo different datasets with hybrid conditioning way.
Hutipost
Yearbook
RMNIST
IMoW
2-Moons
ONP
Inpul shape
x 128 x
172
128 3
2 32
10
128 X
62
256
128 x
2 128 x
Diffusion steps
Noise schedule
linear
lincar
lincar
lincar
lincar
lincar
lincar
Depth
Channel Multiplier
1.2.4
1.2.2
1,2,4
1.2
1,2,4
1.2
Atlention Resolutions
4,2,1
4,2,1
4,2.1
2.1
4,2.1
2.1
Head Channels
Transformer Depth
Batch Size
32
32
32
Learning Rate
8e-5
8c-5
Sc-4
Se-d
8c-5
Sc-4
Se-4
Optimizer
AdamW
AdamW
Adam
AdamW
AdumW
AdamW
Adam
Table 8: Training details on different datasets.
Dataset
Epochs
JDM
Optimizer
Learning Rate
A
L
M
Mg
Huffpost
50
0.6
20
Adam
2e-5
10
32
32
Arxiv
64
5
0.2
5
Adam
2e-5
10
8
32
32
Yearbook
64
50
0.2
5
Adam
lc-3
10
32
32
R.NIST
50
0.2
5
Adam
1e-3
10
32
32
fMoW
64
25
0.6
30
Adam
2e-4
10
8
32
32
2-Moons
64
150
0.2
10
Adam
lc-3
10
32
32
ONP
50
0.2
10
Adam
1e-4
10
32
32
D.2
Network Details
For the backbone the task model, Huffpost and Arxiv apply pretrained DistilBERT base model [33 along with bottleneck layer (20) to reduce the feature dimensions into 128. The bottleneck
layer is implemented the combination of linaer layer, BatchNorm and ReLU. Yearbook uses the 4-layer convolutional network in [48), RMNIST adopts the ConvNet in [29]. and fMoW employs the
DenseNet-122 15] along with a bottleneck layer (00 reduce the feature
dimensions into 256. Meanwhile, 2-Moons uses MLP with two hidden layers of hidden size 64 and
128, and ONP adopts a MLP with one hidden layer hidden size 128.
For the conditional diffusion model, we implement it in U-Net architecture similar to LDM [31 and make modifications better suit our method. Detailed modifications can be found in the code provided in the supplementary material. In Table 7 we provide detailed configurations for U-Net &g on different datasets. Please refer original paper 31 for the meaning different hyperparameters.
D.3 Training Recipe
Training details on different datasets are given in Table where B is the batch size for the task model, IdM is the inner iterations for updating Eg. A is the loss tradeoff hyperparameter, is the warm-up hyperparameter, L the maximum length of the reference point queue Qr. M the maximum length of the anchor point queue Qa and prototype queue Qp. and M, is the number of generated residual classifier weights based on each reference point. All experiments are conducted using the Py PyTorch packages and on single NVIDIA GeForee RTX 4090 GPU with 24GB memory.
E More Results
E.1
Hyperparameter Sensitivity
In Fig. 4(a) we test the sensitivity of W-Diff to the loss tradeoff hyperparameter X, the maximum length L of reference point queue Q, and the number M, generated residual classifier weights based on each reference point, where E (0.1, 1.0, 5.0, 10.0, 50.0). L. E 21,2, 4, 8). M, E (8, 16, 32, 64, 128). We find that larger M, results more weights for ensemble and seems to be better. W-Diff is a little bit sensitive and L. Empirically, À 10.0 and larger work well.
18

---

97.5
97.5
97.5
0.025
RMSNIT
3 92.55
92.5
92.5
0.020
Hulfpost
90.0
90.0
90.0
00
85.0
0.005
82.5
82.5
62.5
80.0
0.1
50
80.0
80.0
0.000
20
Mg
Iterations (x10*)
(a)
Hyperparameter sensitivity.
(b) Loss curve of Casss
Figure 4: (a): Sensitivity of W-Diff to hyperparameters X, L, M, on RMNIST. (b): The loss curve of Cass on RMNIST and Huffpost when training conditional diffusion model on the second domain.
Table 9: Memory and inference time of diffusion model on different datasets.
Yearbook RMNIST IMoW Hulfpost Arxiv 2-Moons ONP
Number of parameters (MB) for the conditional diftusion model Eg
2.31
41.62
27.23 41.62
20.01
10.52 15.51
Time (s) for generating M, = 32 residual classifier weights in a batch,
where denoising step 1000
12
23
71
24
181
21
16
E.2
Convergence of Diffusion Model Training
In Fig. 4(b) we plot the loss curve of Cass when the conditional diffusion model is incrementally trained on the second source domain. Form the results,
that the noise estimation error loss flirs steadily decreases and finally converges, demonstrating that using the FIFO queue to cache recent M classifier weights and prototype matrices after the warm-up stage feasible for the diffusion model training. Storing all checkpoints after the warm-up stage provides lost of training data for diffusion model but requires more storage cost. By contrast, using a FIFO queue with fixed length balances the storage cost and the diversity of training data.
E.3 Memory and Time Cost of Diffusion Model
Table 9 we the model size and inference time of the conditional diffusion model on different datasets. Since the diffusion model only models the evolving pattern of the classifier weights which are the parameters
linear layer, the model size that is measured by the number of parameters small all datasets. Besides, the inference time when forwarding the diffusion model for 1000 times to generate a batch of residual classifier weights moderate. Concretely, forwarding the condition diffusion model for one time requires < 181 ms. Certainly, acceleration techniques, c.g., DDIM 35 can be used to further reduce the inference time.
E.4 Significance test (t-test) W-Diff.
W-Diff vs. EvoS
Jog(0.05)
Yearbook
To comprehensively evaluate the effectiveness of W-Diff, we conduct the significance test (t- test) Huffpost, Arxiv, Yearbook. RMNIST and fMoW datasets. Concretely, a significance level of 0.05 is applied. If the p-value is less than 0.05, then the accuracy difference between EvoS (45 and W-Diff is statistically significant. For clearer explanation, the -log(p) of each p- value is plotted. In Fig. 5] the majority of the -log(p) of the performance comparison between EvoS [45) and W-fiff are larger than -lo5(0.05), which means that W-Diff is statistically superior EvoS 45 at most datasets.
Huffpost
RMNIST
Anxiv
rMoW
20
10
O00 avg.
worst
Dr-
B
pr
wore
pr=
ooD vo
wora
ooD
sop
doo
ooo
doo
cop
Figure 5: t-test for W-Diff vs EvoS
[45),
where
significance level of 0.05 adopted.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have stated the contribution of our work in the introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of the work in Section A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
Our work is a methodological level design and does not propose new theories.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: Yes]
Justification: We have provided the implementation details in Section D and provide the code in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification: The datasets used in our paper are from previous work and are publicly available. We have provided the dataset details in Section Dandd the URL for downloading in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ce.
httpl//guuudes/oooeSuuiiiaassss oos.ssooisssoooooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips. httppssccppblccccucccccoo wetggdddeeeeeeebaiiiiiiiia/a SoreddeubmissionPoo. Policy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have described the problem setting in Section 3.1 and the experimental details in Section 5.1 and D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Ys]]
Justification: We have reported the mean and standard deviation in the main experiments when independently running each task with three random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have pointed out the used GPU in Section and the number parameters for the conditional diffusion model in Section E.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httngh/hhh//////rroo//oo///////a htcc/pubblc//thiiccuilllelellell
Answer: [Yes]
Justification:
We have preserve anonymity in all submitted materials.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the potential social impacts in Section A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The datasets and models used in the paper are all public and from previous works.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The datasets and models used in the paper are from previous works and are public. We have cited related papers in our work and provided the license in Section D.I
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets are released.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Neither crowdsourcing nor research with human subjects is used.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "M2SmKl4h2Q", "reviewer": "Reviewer_EUGW", "summary": "This paper deals with the problem of evolving domain generalization in non-stationary environments, where dynamically changing source domains arrive sequentially, but we only have access to data samples from the current domain (and not the past ones). The main idea is to learn a conditional diffusion model that predicts how the evolving pattern of the classifiers, as the source domains keep changing. In a nutshell, the conditional diffusion model learns how to go from past classifier weights (reference points) to the weights of the current classifier (anchor point), conditioned on prototypes of the current domain. One challenge concerns the shared feature encoder: if the encoder is updated using the latest source domain, then it will overfit to it. To avoid this, the authors suggest to enforce prediction consistency among multiple classifiers, so that all classifiers from past and present domains give the same prediction given the same feature representation. This implies that the past reference points do not become obsolete. During inference, the goal is to predict the classifier weights given the current domain prototypes. For this purpose, the framework uses the conditional diffusion model to cheaply generate a large number classifiers, which it then averages for improved robustness. Extensive experiments on synthetic and real datasets show strong performance for the proposed framework.", "questions": "- Why is it not necessary to condition on the timestep difference between the reference point and the current anchor? Intuitively, one might expect that knowing how far into the past the reference point is could lead to better predictions. Did the authors try this?\n- I was under the impression that U-Net still needs the parameters $\\beta_s$ in Equations (3) and (4).. If yes, how did the authors set these hyperparameters? Where they fixed throughout training?\n- I am not sure I can understand Figure 3a. $\\tilde{W}^{7\\mid 5}$ is concentrated in the upper left, whereas $W^7$ is at the very bottom. And, yet, both of them perform very well in Figure (3b). What does that exactly say about the computed weights? Maybe it means that the visualization does not really tell us anything about test performance?\n- In the ablation study in Section 5.3, what is Variant B?\n- Can the authors be more clear about the limitations of their framework? Could their framework be modified to also predict the future data distribution (e.g., how the $x_i$ will be distributed in the next timestep)?", "limitations": "No concern.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Overall, the framework is interesting. The use of conditional diffusion models makes sense in thee context of non-stationary adaptation, as such models can capture the dynamically changing source domains. Furthermore, it improves robustness because it is possible to generate multiple weight predictions, and then take their average.\n- The consistency constraint is a nice way to force the shared representation that remains valid for old classifiers, even as the source domain keeps changing. The use of reference points makes it possible to completely get rid of old data (except the data related to the current source domain), and is efficient. The diffusion models are able to capture even complex non-stationarities, which gives the model significant expressive power.\n- The experimental study is quite extensive and shows strong performance across a variety of benchmarks for the proposed weight-diffusion framework.", "weaknesses": "- In inference phase, the authors assume they know the dataset (without the labels) for the future timesteps $\\{T+1,\\dots,T+K\\}$. Of course, this cannot work when the datasets are not given. In such a case, the model must also be in a position to infer the future data distribution for the points $x_i$, in order to then estimate the prototypes $f_{test}$. The current framework seems not to be able to deal with that. I think it would help if the authors explain what the limitations of their framework are, and why for instance it is very difficult to predict the future data distribution in non-stationary environments. The current weight-diffusion model assumes that we can have access to future data but not their labels during inference, but this may not always be the case.\n- I like the fact that the authors condition on the context. But I was not clear why the context only consists of the historical weights and the current prototypes. Why not also include the timestep difference between the reference point and the current timestep? For instance, assume we are given the context (reference point + prototype). If the reference point is from 5 timesteps ago, then the predicted weights might be different compared to the case where the reference point is from 10 timesteps ago, even if the reference point value is the same. If we are not given how far into the past the reference point lies, then I was thinking that predicting the current weights is harder. On the other hand, perhaps this is already taken into account implicitly, in the framework, but this is not immediately clear.\n- The point above also points to the limitation of the current framework, which is a lack of formal theory. It works very well  experimentally, but some aspects are not very clear.\n- There are numerous typos and bad grammar throughout the paper. The authors should do a very careful proofreading and fix all the errors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your efforts in reviewing the paper. Below, we respond to your questions in detail.\n\n> **Q1: Concerns about the assumption of dataset accessibility in the inference phase.**\n\nThanks. Firstly, we would like to make it clear that for discriminative tasks, test data (i.e., target data in our paper) must have been given, when the model is used for testing. Thus, our method doesn't need to infer future data distributions. We can estimate the prototype matrix directly by feeding test data into the model to get their features and class probability predictions.\n\nSecondly, previous evolving domain generalization (EDG) works [4, 6, 7, 11] are only evaluated on the next one target domain. Following [8, 10], we evaluate on the next $K$ target domains, hoping models can also generalize well on the target domain in the farther future. When $K=1$, it aligns with previous standard evaluations. Besides, the evaluation on each domain is conducted independently.\n\nFinally, given that the whole data of a target domain may not come at once, we also offer a way for estimating the prototype matrix in a batch-data stream scenario, where the prototype matrix for the $j$-th target data batch is estimated using the cumulative moving average. In **Fig. 1(a)** of the PDF file, we provide the results on RMNIST and Huffpost when evaluating W-Diff in the batch-data stream scenario.\n\n\n> **Q2: Why not include timestamp difference into the condition?**\n\nThanks. In our method, timestamp difference information is implicitly considered, as domains evolve over time and arrive in chronological order during training. Besides, the difference between classifier weights of current and historical domains also contains implicit timestamp difference information. Moreover, due to different optimization starting points and stochastic gradient descent, it is unlikely that classifiers respectively trained for two domains are exactly the same. Hence, we do not explicitly include timestamp difference in the condition.\n\nNevertheless, as you suggested, we have also tried to explicitly include the timestamp difference $\\Delta t$. Results on RMNIST are shown in **Table 1** of the PDF file, where no obvious improvement is obtained, possibly due to reasons above.\n\n> **Q3: The framework works very well experimentally but lacks formal theory.**\n\nThanks. The gradient descent algorithm (GDA) is known to be theoretically supported, and we find the denoising process of diffusion models can be seen as a learnable GDA with momentum update and small noise perturbation. Let us consider the following gradient descent algorithm for optimizing the model with parameters $\\Phi$ on dataset $D$ via minimizing loss $L$:\n\n$\\Phi_{i+1}=\\Phi_i-\\lambda \\cdot \\nabla_{\\Phi}L(\\Phi_i, D),\\quad i=0,\\ldots,T-1,$\n\nwhere $\\lambda$ controls the update step size.\n\nAlternatively, we generate classifier weights via conditional diffusion model $\\mathcal{E}_{\\theta}$ with the following denoising process:\n\n$W^t\\_{s-1}=\\frac{1}{\\sqrt{\\alpha_s}}\\left(W^t\\_s - \\frac{\\beta\\_s}{\\sqrt{1-{\\bar{\\alpha}}\\_s}} \\mathcal{E}\\_\\theta(W^t\\_s,s,\\ddot W^{t^{\\prime}}, \\mu^t)\\right) + \\sigma\\_s \\epsilon, \\quad \\epsilon \\sim N(\\mathbf 0,\\mathbf I),$\n\nwhere $s$ is diffusion step, $W^t_0$ is the overfitted classifier weights on the $t$-th domain $D^t$, $\\mu^t$ is the prototype matrix of $D^t$, and $\\ddot W^{t^{\\prime}}$ is the saved overfitted classifier weights on historical domain $D^{t^{\\prime}}, t^{\\prime} < t$.\n\nThen, we can reformulate the denoising process as\n\n$W^t\\_{s-1}=W^t\\_s-\\underbrace{\\frac{\\beta\\_s}{\\sqrt{\\alpha\\_s}\\sqrt{1-{\\bar{\\alpha}}\\_s}}}\\_{\\lambda} \\mathcal{E}\\_{\\theta}(W^t\\_s,s, \\ddot W^{t^{\\prime}}, \\mu^t)+\\underbrace{(\\frac{1}{\\sqrt{\\alpha\\_s}}-1)W^t\\_s}\\_{\\text{momentum update}}+\\underbrace{\\sigma\\_s \\epsilon}\\_{\\text{noise perturbation}}.$\n\nCompared with GDA, the conditional diffusion model can be viewed as learning a descent path of $\\nabla_W L(W^t_s, D^t|\\ddot W^{t^{\\prime}}, \\mu^t)$, so that conditioned on historical classifier weights and current prototype matrix, the diffusion model can directly generate classifier weights suitable for current domain.\n\n> **Q4: Typos and grammar errors.**\n\nThanks. We have proofread the paper carefully and corrected the errors.\n\n> **Q5: Settings of $\\beta_s$ in U-Net.**\n\nThanks. Eq.(3) is obtained by using $\\alpha_s=1-\\beta_s$. And $\\beta_s$ is set via the following code, and betas is fixed throughout training.\n\n```\nbetas=(torch.linspace(1e-4 ** 0.5, 2e-2 ** 0.5, 1000) ** 2)\n```\n\n> **Q6: Explanation of Figure 3a.**\n\nThanks. What we want to convey is that our generated classifier weights are diverse and generally high-performance. $W^7$ is obtained by fine-tuning the classifier on domain $\\mathcal{D}^7$, which may be trapped in local optima. The parameter space is too big to fully explore during fine-tuning. It does not mean that only the area around $W^7$ is good. $\\hat W^{7|5}$ generally performs well, meaning that our generated classifier weights are diverse and potentially cover better but unexplored area in the fine-tuning process.\n\n\n> **Q7: What is Variant B in ablation study?**\n\nThanks. Variant B ablates the conditional diffusion model and directly uses the incrementally trained classifier along with the learned domain-shared feature encoder for inference.\n\n> **Q8: Could it be modified to predict future data distribution?**\n\nThanks. Predicting future data distribution is possible by saving partial historical data to learn the evolving pattern at the data level, but it is not wise for discriminative tasks (e.g., classification), which requires further fine-tuning on generated instances to generalize discriminative models to target domain. Moreover, due to varied data types (e.g., image, text, multivariate in our experiments), data diffusion is cumbersome, which requires quite different architectures and lacks of universality. By contrast, our weight diffusion is more general for discriminative tasks on different datatypes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J86hh372MN", "reviewer": "Reviewer_tBNz", "summary": "This paper presents Weight Diffusion (W-Diff), a framework for domain generalization in non-stationary environments. W-Diff leverages a conditional diffusion model in the parameter space to learn the evolving pattern of classifiers during domain-incremental training. Experiments on synthetic and real-world datasets demonstrate its superior generalization performance on unseen future domains", "questions": "Please refer to my weaknesses section.", "limitations": "The authors discussed some limitations of this work in their paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**S1.**\nThis paper introduces a novel approach for capturing evolving patterns at the parameter level using diffusion models.\n\n**S2.**\nThe proposed method demonstrates good performance in generalizing to unseen domains on diverse datasets.\n\n**S3.**\nThis paper addresses the practical challenge of sequentially arriving non-static source domains, mimicking real-world scenarios.", "weaknesses": "**W1.**\nThe paper does not sufficiently justify why a diffusion model is chosen for the domain generalization (DG) problem. While diffusion models have shown excellent performance in generation tasks, the specific advantages they offer for DG are not clearly articulated.\n\n**W2.**\nThe computational complexity of the diffusion model training might be a barrier for very large datasets or real-time applications. A thorough analyses on the computational complexity espeically for training is needed.\n\n**W3.**\nThe evaluation is limited to classification tasks; applicability to other types of tasks, such as regression, remains unexplored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your efforts in reviewing the paper as well as your constructive comments. Below, we do our utmost to address your concerns.\n\n> **Q1: The specific advantages that diffusion models offer for DG.**\n\nThanks for your comment. Firstly, instead of learning a deterministic classifier like previous EDG methods [4, 8], we consider the classifier as a distribution, which conduces to improving the robustness and reducing miscalibration. Yet, the prior knowledge of distribution type is unknown and we only have the observed data (i.e., saved classifier checkpoints) from the unknown distribution. Fortunately, diffusion models are quite powerful in modeling complicated and unknown distribution based on observed data. Moreover, we have also tried other generative models, e.g., VAE, to generate the classifier weights. The results on RMNIST dataset are shown in the following table. We see that the generalization performance is worse when using VAE, which demonstrates the superiority of diffusion model in modeling complicated classifier distributions.\n\n-----Accuracy (%) on RMNIST (K = 3)-----\n|method|generative model|$\\mathcal{D}^{T+1}$|OOD avg.|OOD worst|\n|:---:|:---:|:---:|:---:|:---:| \n|W-Diff|VAE| 98.66|93.83|87.16|\n|W-Diff|diffusion model|**98.70**|**94.12**|**87.36**|\n\nSecondly, in the paper, we focus on the problem of evolving domain generalization (EDG), where the domain gradually evolves over time in an underlying pattern. In addition to the invariant feature learning, it is also critical to excavate the underlying evolving pattern across domains for better predicting the model status on future domains. Inspired by the fact that conditional diffusion models excel at generating specific images using additional information, we delicately design the condition including both the classifier weights of historical domain and the prototypes of current domain, and train the conditional diffusion model to generate the discrepancy of classifier weights between historical domain and current domain. The discrepancy represents the evolving of classifier across domains. In such way, we convey the information on classifier evolving from past to present to the conditional diffusion model.\n\t\nThirdly, during inference, we can inject the target data information through the prototypes in the condition to generate target-customized classifiers. Besides, since the diffusion model is stochastic in nature, multiple noise samplings can generate different classifiers, forming an ensemble, which leads to better robustness for generalizing on new domains.\n\n> **Q2: The computational complexity of the diffusion model training.**\n\nThanks for your advice. For simplicity, we take the U-Net with all convolutional layers as an example. Assuming that there are $L$ convolutional layers, the size of feature map in the $i$-th layer is $H_i \\times W_i×C_i$ and the kernel size is $k_i\\times k_i$. Then the time complexity of one forward pass is $\\mathcal{O}(\\sum_{i=1}^L H_i \\times W_i \\times C_i \\times C_{i-1} \\times k_i^2)$. Let $S$ denote the total time step in diffusion model. Then the time complexity of training diffusion models for $I$ iterations can be approximated as $\\mathcal{O}(I \\times S \\times \\sum_{i=1}^L H_i \\times W_i \\times C_i \\times C_{i-1} \\times k_i^2)$.\n\nMoreover, in **Table 5** of the PDF file, we compare the training time and GPU memory of our W-Diff, DRAIN [4], EvoS [8] and GI [11] on the RMNIST and Huffpost datasets. We acknowledge that our method has no significant advantage in terms of training time, due to the diffusion model. But this is not a limitation unique to our approach and most methods based on diffusion models have this limitation. As part of future work, we will investigate and try to address the limitation to further enhance the training efficiency.\n\nIn addition, it is worth mentioning that GI and DRAIN require huge computational resources during the training process, when they are applied on relatively large networks. Specifically, DRAIN needs to generate the entire network parameters and the fine-tuning stage of GI requires second-order gradients. On the Huffpost dataset with the backbone of DistilBERT-base and a batch size of 64, GI and DRAIN encounter the issue of **GPU memory explosion**. By contrast, our method utilizes diffusion model to generate only classifier weights, and as shown in the following table, the diffusion model is small to train, without the explosion of GPU memory when using the same batch size of 64. Overall, the main contribution of our work is providing a new perspective to address EDG in the domain-incremental setting via delicately tailoring the conditional diffusion model.\n\n-----------------Number of parameters (MB) of conditional diffusion model $\\mathcal{E}_{\\boldsymbol{\\theta}}$----------------\n|Yearbook|RMNIST|fMoW|Huffpost|Arxiv|2-Moons|ONP|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:| \n|2.31|41.6|27.23|41.62|20.01|10.52|15.51|\n\n\n> **Q3: The evaluation is limited to classification tasks; applicability to other types of tasks, such as regression, remains unexplored.**\n\nThanks for your advice. We have extended our method to the two regression datasets (i.e., House and Appliance) in DRAIN [4]. For regression tasks, the prototype is calculated at the domain level, i.e., the average of features in a single domain. And the cross-entropy loss is replaced with the mean squared error (MSE) loss and the Kullback-Leibler divergence in the consistency loss is replaced with MSE. Specifically, the results are provided in the following table, where our method still achieves better generalization performance.\n\n-----------------Mean absolute error (MAE) for regression tasks----------------\n| Method | House | Appliance |\n|:---: | :---: | :---:|\n|Offline |11.0$\\pm$0.36|10.2$\\pm$1.1|\n|IncFinetune |9.7$\\pm$0.01|8.9$\\pm$0.5|\n|CIDA |9.7$\\pm$0.06|8.7$\\pm$0.2|\n|GI |9.6$\\pm$0.02|8.2$\\pm$0.6 |\n|DRAIN |9.3$\\pm$0.14|6.4$\\pm$0.4 |\n|**W-Diff** |**9.1$\\pm$0.15** |**4.9$\\pm$0.3**|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vR4BSUCHPj", "reviewer": "Reviewer_Ve2z", "summary": "The paper proposes a novel method called Weight Diffusion (W-Diff), which employs a conditional diffusion model in the parameter space to learn the evolving patterns of classifiers during domain-incremental training. During inference, the proposed method uses an ensemble of classifiers tailored to the target domain for robust predictions.", "questions": "See Weaknesses.", "limitations": "The limitation of the paper is that it only considers generating the classifier. It would be interesting to explore whether it can generate the entire network.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work is pioneering in applying diffusion models for generating parameters in a practical context.\n2. The paper is well-written with detailed descriptions of each component of the algorithm.\n3. The paper demonstrates the effectiveness of W-Diff through comprehensive experiments on both synthetic and real-world datasets.\n4. The use of an ensemble of classifiers enhances prediction robustness.", "weaknesses": "1. The experiments are conducted on relatively small networks, and the diffusion model is used solely to generate the classifier. It remains unclear whether this method can scale to larger networks.\n2. The paper should include a comparison with Variational Autoencoders (VAEs) or other generative models. Given that diffusion models are more complex and challenging to train, it is important to demonstrate the necessity of using them.\n3. The paper should include a comparison of training time with state-of-the-art algorithms.\n4. It would be good to include a notation table in the appendix, as there are numerous notations used throughout the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your efforts in reviewing the paper and the constructive comments. Below, we have tried our best to address your concerns in detail.\n\n> **Q1: Can this method scale to larger networks and generate the entire network?**\n\nThanks. Firstly, we have tried larger networks on the fMoW dataset by replacing the DenseNet-121 with DenseNet-161, DenseNet-169, DenseNet-201, respectively. The results are provided in **Table 2** of the PDF file, where further performance improvements are obtained. \n\nSecondly, our method can be extended to generate more parameters, but some modifications are needed to solve the different shapes of parameters from multiple layers and the training efficiency of diffusion models when the number of parameters to be generated is high. One possible solution to the shape difference is to flatten parameters from different layers and then concatenate all the flattened parameters into a vector. The generated parameters in a vector format are finally converted into their original shapes. As for the training efficiency, we can additionally train a VAE to encode the high-dimensional parameter vector into a low-dimensional embedding and then decode the embedding to reconstruct the parameter vector. Later, the diffusion model is trained in the low-dimensional embedding space and the decoder of VAE is used to recover parameters from the generated embedding via diffusion model in the inference stage. As stated in the Limitations section, we will leave this for a future work due to time constraints.\n\nThirdly, as mentioned above, generating the entire network is possible, but it is not a good choice for evolving domain generalization (EDG).\n\n1) The massive parameters of the entire network require a large number of sequential source domains to accurately model the evolving pattern of the entire network.\n\n2) Not all parameters are unshared. Shallow layers of the network usually extract general knowledge that can be shared, while deep layers are specific to tasks or datasets [5].\n\n3) Previous EDG work [7] has provided theoretical evidence, showing that solely learning dynamic features is insufficient. [6] and [7] both use a static variational encoding network as the feature encoder to extract invariant features and train a domain-adaptive classifier. Hence, based on the experience and conclusion of previous works, we choose to only generate the parameters of task head, e.g., the classifier for classification tasks.\n\n4) Moreover, in the paper, we compare our W-Diff with DRAIN [4] which leverages LSTM to learn the evolving pattern of the entire network. Experimentally, on 2-Moons and ONP datasets, the error rate of target domain is **3.2% (DRAIN) vs. 1.5% (W-Diff), 38.3% (DRAIN) vs. 32.9% (W-Diff)**. These results partially show that generating entire network is less effective for EDG.\n\n\n> **Q2: Comparison with VAEs.**\n\nThanks for your advice. We have added the results when replacing the diffusion model with VAE to generate classifier weights, conditioned on the reference point and prototypes. Please refer to **variant F** in **Table 4** of the PDF file. The better generalization performance when using diffusion model to learn the evolving pattern of classifiers shows its superiority in modeling complex distributions and generating high-quality data, which has also been proven by many diffusion-based generation works. Besides, to reduce the difficulty of training diffusion models, we only generate the parameters of the classifier. The number of parameters (MB) for the conditional diffusion model is given in **Table 6** of the PDF file, from which we see that the diffusion model is small and easy to train.\n\n\n> **Q3: Comparison of training time with SOTA method.**\n\nThanks. The training time complexity of our method mainly comes from the diffusion model. For simplicity, we take the U-Net with all convolutional layers as an example. Assuming that there are $L$ convolutional layers, the size of feature map in the $i$-th layer is $H_i \\times W_i×C_i$ and the kernel size is $k_i\\times k_i$. Then the time complexity of one forward pass is $\\mathcal{O}(\\sum_{i=1}^L H_i \\times W_i \\times C_i \\times C_{i-1} \\times k_i^2)$. Let $S$ denote the total time step in diffusion model. Then the time complexity of training diffusion models for $I$ iterations can be approximated as $\\mathcal{O}(I \\times S \\times \\sum_{i=1}^L H_i\\times W_i \\times C_i\\times C_{i-1}\\times k_i^2)$.\n\nMoreover, in **Table 5** of the PDF file, we compare the training time and GPU memory of our W-Diff, DRAIN [4], EvoS [8] and GI [11] on the RMNIST and Huffpost datasets. We acknowledge that our method has no significant advantage in terms of training time, due to the diffusion model. But this is not a limitation unique to our approach and most methods based on diffusion models have this limitation. As part of future work, we will investigate and try to address the limitation to further enhance the training efficiency.\n\nIn addition, it is worth mentioning that GI and DRAIN require huge computational resources during the training process, when they are applied on relatively large networks. Specifically, DRAIN needs to generate the entire network parameters and the fine-tuning stage of GI requires second-order gradients. On the Huffpost dataset with the backbone of DistilBERT-base and a batch size of 64, GI and DRAIN encounter the issue of **GPU memory explosion**. By contrast, our method utilizes diffusion model to generate only classifier weights, and as shown in **Table 6** of the PDF file, the diffusion model is small to train, without the explosion of GPU memory when using the same batch size of 64. Overall, the main contribution of our work is providing a new perspective to address EDG in the domain-incremental setting via delicately tailoring the conditional diffusion model.\n\n\n> **Q4: Include a notation table in the appendix.**\n\nThanks for your advice. We will add a notation table in the revision to ease the burden of readers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9MprPdUj7Y", "reviewer": "Reviewer_BuMk", "summary": "This manuscript aims to tackle the evolving domain generalization (EDG) issue, namely the domain gradually evolves in an underlying continuous structure. The paper introduces the idea of Weight Diffusion (W-Diff), a conditional diffusion model in the parameter space to learn the evolving pattern of classifiers. Combining such types of classifier with weight ensembling and a domain-shared feature space allows robust prediction. The effectiveness of the proposed method is examined on two text classification datasets, three image classification datasets, and two multi-variate classification datasets.", "questions": "NA", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The evolving domain generalization is interesting yet important to the research community.\n* The manuscript is well-structured. It explains its methodology design clearly and intuitively.\n* The manuscript introduces the idea of model weight generation through the diffusion model to the area of evolving domain generalization. The concept itself is interesting. However, the authors still need to discuss the related work carefully.\n* The proposed method is examined on two text classification datasets, three image classification datasets, and two multi-variate classification datasets.", "weaknesses": "* Unclear novelty. The idea of using a diffusion model to generate model weights/heads appeared in [1, 2]. However, the manuscript did not cite and discuss these two papers (and maybe their follow-up works), making the exact contribution unclear. From the reviewer's perspective, idea 1 of using the conditional diffusion model to model parameter evolution patterns is interesting and intuitive; idea 2 of learning domain-shared feature encoder is standard. Combining idea 1 and idea 2 and applying it to the area of evolving domain generalization is ok, but still needs a careful discussion and ablation study.\n* The performance gain over different datasets looks marginal.\n* The manuscript needs more ablation studies. E.g., What if instead of generating the classifiers on the fly for the unseen domain, we just leverage the past classifiers for the ensemble prediction, or test-time adaptive classifier ensembling as in [3]?\n\n\n### Reference\n[1] Learning to Learn with Generative Models of Neural Network Checkpoints. https://arxiv.org/abs/2209.12892\n\n[2] Diffusion-based Neural Network Weights Generation. https://arxiv.org/abs/2402.18153\n\n[3] Adaptive Test-Time Personalization for Federated Learning. https://arxiv.org/abs/2310.18816", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Sincerely thanks for your efforts in reviewing the paper. Below, we respond to your questions in detail.\n\n> **Q1: Discussion with the two related papers [1, 2].**\n\nThanks. Our method differs from [1, 2] as follows:\n\n**Focused problem**: G.pt [1] focuses on supervised learning and reinforcement learning, while our W-Diff addresses the evolving domain generalization (EDG) in the domain-incremental setting. In EDG, distribution shifts often hinder models that are trained with full supervision on in-distribution (ID) data from generalizing to out-of-distribution data. Hence, our goal is to address the distribution shift, instead of generating diverse high-performance parameters for ID data.\n\nD2NWG [2] focuses on transfer learning to provide better model parameter initialization for faster fine-tuning convergence on new datasets. However, target domains are unlabeled in EDG, preventing supervised fine-tuning. This makes D2NWG unsuitable for domain generalization. In contrast, our method generates model parameters **applicable directly to the unlabeled target domain without fine-tuning**.\n\n**Condition design for diffusion model**: G.pt [1] collects the loss/error/return of task model checkpoints during training as the condition for the diffusion model. It is designed for a single dataset to which the training data belongs, thus struggling with distribution shifts.\n\nD2NWG [2] uses CLIP to extract features for each sample and Set Transformer to generate dataset encoding from these features. The dataset encoding is used as the condition for diffusion model, while training set samples of a new dataset are required to obtain the dataset encoding. This is infeasible in unlabeled target domains.\n\nDifferent from [1, 2], we use the classifier weights of historical domain (referred as reference point) and the prototypes of current domain as the condition of diffusion model to generate the difference of classifier weights between reference point and anchor point (i.e., the classifier weights of current domain). Considerations of such design are 1) the difference between reference point and anchor point denotes the evolution of parameters from historical to current domain, which **conduces to modeling the crucial evolving pattern across domains in EDG**; 2) the reference point provides initialization-like information, while current prototypes offer some information about the desired decision boundary, **helping to explore the relationship between generated parameters and the given domain**.\n\nMoreover, we compare W-Diff with G.pt [1] in the following datasets. G.pt shows worse generalization on target domain, due to the limitations discussed above.\n\n----Error rate comparison (%)---\n|Method|2-Moons|ONP|\n|-|-|-|\n|G.pt|4.5$\\pm$1.2|35.1$\\pm$0.8|\n|**W-Diff**|**1.5$\\pm$1.0**|**32.9$\\pm$0.5**|\n\n> **Q2: The idea is ok, but still needs a careful discussion and ablation study.**\n\nThanks. We have provided the results of ablation study in **Table 4** of the PDF file. Concretely, we explore the following variants:\n\n* Variant A ablates the consistency loss $\\mathcal{L}^t_{con}$ for learning domain-shared feature encoder.\n\nThe performance drop of Variant A suggests that learning domain-invariant feature representations is necessary for EDG in the domain-incremental setting. Otherwise, the feature encoder could easily overfit to current domain, prohibiting task model from generalization.\n\n* Variant B ablates the conditional diffusion model and directly uses the incrementally trained classifier for inference.\n\n* Variant C directly uses the historical classifier weights in the reference point queue $Q_r$ to construct the average weight ensemble $\\bar{\\mathbf{W}}^{test}$ for inference: $\\bar{\\mathbf{W}}^{test}=\\frac{1}{|Q_r|}\\sum_{\\ddot{\\mathbf{W}}^{t^{\\prime}}\\in Q_r}\\ddot{\\mathbf{W}}^{t^{\\prime}}$.\n\nThe inferior results of variant B and C indicate that W-Diff benefits from generating meaningful and customized classifier weights via controlling the condition of diffusion model.\n\n\n> **Q3: The performance gain over different datasets looks marginal.**\n\nThanks. To comprehensively evaluate the effectiveness of W-Diff, we have conducted a significance test (t-test) on different datasets. Concretely, a significance level of 0.05 is applied, and if p-value is less than 0.05, the accuracy difference between EvoS [8] and W-Diff is statistically significant. For clearer explanation, the -log(p) of each p-value has shown in red line. In **Fig. 1(b)** of the PDF file, majority of the -log(p) of the performance comparison between EvoS and W-Diff are larger than -log(0.05), which means W-Diff is statistically superior to EvoS at most datasets.\n\n\nBesides, we also extend W-Diff to regression tasks, where the regression datasets and results are from DRAIN [4]. In **Table 3** of the PDF file, W-Diff still works well. Overall, given the generalization performance on regression and text/image/multi-variate classification datasets, W-Diff is of more versatility.\n\n> **Q4: More ablation studies, e.g., just leveraging the past classifiers for the ensemble prediction, or test-time adaptive classifier ensemble as in [3]?**\n\nThanks. As you suggested, we have added more ablation study results in **Table 1** of the PDF file.\n\n* Variant C in the reply to **Q2** denotes leveraging the past classifiers for the ensemble prediction.\n\n* Variant D denotes using the APT-batch manner in [3], which first conducts unsupervised adaptation of classifier by back-propagating the gradient of entropy loss to source-trained classifier and then makes independent predictions on each batch.\n\n* Variant E denotes using the APT-online manner in [3], which first updates source-trained classifier with the cumulative moving average of update directions from the first target data batch to current batch, and then makes predictions on current batch.\n\nIn **Table 1** of the PDF file, W-Diff outperforms these variants, because they ignore the evolving pattern which is crucial for EDG.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Sincerely thank all the reviewers for their efforts in reviewing our paper and providing constructive suggestions. We are greatly encouraged that the reviewers find that

* our framework/idea is **interesting** (*Reviewer EUGW and BuMk*), **novel** (*Reviewer tBNz*), and **pioneering** in applying diffusion models for generating parameters in a **practical** context (*Reviewer Ve2z*);
* the considered problem of evolving domain generalization is **practical** (*Reviewer tBNz*), **interesting yet important** (*Reviewer BuMk*);
* our paper is **well-written** with detailed descriptions (*Reviewer Ve2z*) and **well-structured** (*Reviewer BuMk*);
* the experimental study is **extensive** (*Reviewer EUGW*) and **comprehensive** (*Reviewer Ve2z*);
* and the performance is **strong** across a variety of benchmarks (*Reviewer EUGW*), **good** in generalizing to unseen domains on **diverse** datasets (*Reviewer tBNz*).

As for the concerns and suggestions raised by each reviewer, we have done our best to address them thoroughly and have provided detailed responses to each of them. Below are references that we used in our replies to each reviewer. Additionally, a `one-page PDF` file that includes the relevant figures and tables referenced in our replies has been uploaded. Please refer to this PDF file for detailed results, if needed.

Refs:

[1] Learning to Learn with Generative Models of Neural Network Checkpoints. arXiv:2209.12892, 2022.

[2] Diffusion-based Neural Network Weights Generation. arXiv:2402.18153, 2024.

[3] Adaptive Test-Time Personalization for Federated Learning. In NeurIPS, 2023.

[4] Temporal domain generalization with drift-aware dynamic neural networks. In ICLR, 2023.

[5] How transferable are features in deep neural networks? In NeurIPS 2014.

[6] Generalizing to evolving domains with latent structure-aware sequential autoencoder. In ICML, 2022.

[7] Enhancing Evolving Domain Generalization through Dynamic Latent Representations. In AAAI, 2024.

[8] Evolving standardization for continual domain generalization over temporal drift. In NeurIPS, 2023.

[9] High-resolution image synthesis with latent diffusion models. In CVPR, 2022.

[10] Wild-time: A benchmark of in-the-wild distribution shift over time. In NeurIPS, 2022.

[11] Training for the future: A simple gradient interpolation loss to generalize along time. In NeurIPS, 2021.

Meta-review: **Summary**

This paper proposes a novel method called Weight Diffusion (W-DiFF) for EDG. The approach leverages a conditional diffusion model to learn the evolving patterns of classifiers during domain-incremental training, allowing the model to generalize more effectively to unseen future domains. The method is validated through extensive experiments across various datasets, including text, image, and multi-variate classification tasks.


**Decision**

The decision to accept is based on the proposed method’s ability to tackle a challenging and practical setting of EDG, where source domains are not simultaneously available. This represents an advancement over existing approaches that typically assume all source domains are available concurrently.  Additionally, the method introduces an innovative application of a conditional diffusion model to learn evolving patterns in classifier weights, which, while subject to some debate among the reviewers, adds a novel perspective to the field of domain generalization. The strong performance across multiple benchmarks, combined with the comprehensive rebuttal that addressed concerns (e.g., computational cost) and provided further validation, supports the acceptance of this paper.

For the final version, please ensure that the additional experiments and comparisons provided during the rebuttal are included, as they significantly strengthen the paper's contribution.
Decision: Accept (poster)