SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\min  _ {\mathcal {E}, \mathcal {C}} \mathcal {L} _ {c e} ^ {t} = \frac {1}{N ^ {t}} \sum_ {i = 1} ^ {N ^ {t}} \mathcal {C E} \left(\boldsymbol {p} _ {i} ^ {t}, y _ {i} ^ {t}\right), \quad \boldsymbol {p} _ {i} ^ {t} = \operatorname {s o f t m a x} \left(\mathcal {C} \left(\frac {\boldsymbol {f} _ {i} ^ {t} - \operatorname {s g} (\hat {\boldsymbol {\mu}} ^ {t})}{\exp (\operatorname {s g} (\hat {\boldsymbol {a}} ^ {t}))}\right)\right), \tag {8}
$$
where  $\mathcal{C}\mathcal{E}(\cdot ,\cdot)$  is the cross-entropy. Here, stopping gradient  $\mathrm{sg}(\cdot)$  is adopted to stabilize training.
Losses  $\mathcal{L}_{stan}^{t}$  and  $\mathcal{L}_{con}^{t,w}$  are responsible for the training of the attention module  $\mathcal{A}_w$  at time point  $t_t$ ,  $w = 1,2,\dots ,W$ . The former loss  $\mathcal{L}_{stan}^{t}$  in Eq. 10 is minimized to ensure that the standardized feature  $\boldsymbol{z}_i^t$  follows a standard normal distribution. Specifically, we first calculate the mean vector and variance vector of the standardized features as
$$
\text {m e a n}: \boldsymbol {\hat {\mu}} ^ {t} = \frac {1}{N ^ {t}} \sum_ {i = 1} ^ {N _ {t}} \frac {\operatorname {s g} \left(\boldsymbol {f} _ {i} ^ {t}\right) - \boldsymbol {\hat {\mu}} ^ {t}}{\exp \left(\boldsymbol {\hat {a}} ^ {t}\right)}, \quad \text {v a r i a n c e}: \boldsymbol {\hat {v}} ^ {t} = \frac {1}{N ^ {t} - 1} \sum_ {i = 1} ^ {N _ {t}} \left(\frac {\operatorname {s g} \left(\boldsymbol {f} _ {i} ^ {t}\right) - \boldsymbol {\hat {\mu}} ^ {t}}{\exp \left(\boldsymbol {\hat {a}} ^ {t}\right)} - \boldsymbol {\hat {\mu}} ^ {t}\right) ^ {2}. \tag {9}
$$
Then the loss  $\mathcal{L}_{stan}^t$  at time point  $\mathfrak{t}_t$  is expressed as
$$
\left\{ \begin{array}{l} \min  _ {\hat {\boldsymbol {\mu}} ^ {t}, \hat {\boldsymbol {a}} ^ {t}} \mathcal {L} _ {s t a n} ^ {t} = \| \hat {\boldsymbol {\mu}} ^ {t} - \mathbf {0} \| _ {2} + \| \hat {\boldsymbol {v}} ^ {t} - \mathbf {1} \| _ {2}, \quad 1 \leq t \leq 2 \\ \min  _ {\mathcal {A} _ {w}} \mathcal {L} _ {s t a n} ^ {t} = \| \hat {\boldsymbol {\mu}} ^ {t} - \mathbf {0} \| _ {2} + \| \hat {\boldsymbol {v}} ^ {t} - \mathbf {1} \| _ {2}, \quad t \geq w + 2 \end{array} , \right. \tag {10}
$$
where  $d_{f}$  is the dimension of features and  $w$  is the length of the sliding time window. Here, we just want the learnable/generated  $\hat{\pmb{\mu}}^t,\hat{\pmb{a}}^t$  to approach the statistics of true feature distribution, so stopping gradient operation  $\mathrm{sg}(\cdot)$  is conducted on  $\pmb {f}_i^t$  to prevent the gradient of  $\mathcal{L}_{stan}^{t}$  flowing to the feature encoder  $\mathcal{E}$ . Otherwise, the gradient of  $\mathcal{L}_{stan}^{t}$  may distort the feature distribution. Ideally, if  $\mathcal{L}_{stan}^{t}$  is minimized, the generated statistics  $\hat{\pmb{\mu}}^t$  and  $\exp (\hat{\pmb{a}}^t)$  by our MSAM would be equal to the mean vector and standard deviation vector of the true feature distribution of  $\mathcal{D}^t$ . This means that our MSAM can predict future feature distributions based on historical feature statistics.
And the loss  $\mathcal{L}_{con}^{t}$  is to ensure the consistency between generated statistic tokens in the next sliding time window and real statistic tokens. Specifically, we calculate  $\mathcal{L}_{con}^{t,w}$  for attention module  $\mathcal{A}_w$  as
$$
\min  _ {\mathcal {A} _ {w}} \mathcal {L} _ {c o n} ^ {t, w} = \frac {1}{w - 1} \sum_ {k = 1} ^ {w - 1} \| \hat {\boldsymbol {s}} _ {w} ^ {t - w + k} - \boldsymbol {s} ^ {t - w + k} \| _ {2}, \quad (t \geq w + 2) \wedge w \geq 2, \tag {11}
$$
where  $\hat{s}_w^j$  and  $s_w^j$ ,  $j = t - w + 1, \dots, t - 1$  are respectively the predicted statistic token for time point  $t_j$  using the attention module  $\mathcal{A}_w$  and the real statistic token from memory pool  $\mathcal{M}$ . This loss is expected to also capture the evolutionary pattern within the time window.
The fourth loss  $\mathcal{L}_{adv}^t$  is introduced with this consideration that the feature extractor  $\mathcal{E}$  may overfit to current domain and lacks generalizability. To avoid this, we additionally conduct adversarial training between the feature encoder  $\mathcal{E}$  and a discriminator  $\mathfrak{D}$ . Nevertheless, historical data are unavailable. So we instead resort to the learned/generated statistics at previous time points. Concretely, we randomly sample a batch of  $B$  samples  $\{\pmb{f}_{1}^{\prime m},\dots ,\pmb{f}_{B}^{\prime m}\}$  from each distribution  $\mathcal{N}(\pmb {\mu}^m,(\exp (\pmb {a}^m))^2)$ ,  $m = 1,2,\dots ,t - 1$ , via Eq. 12 at each iteration, and use them as the proxy of historical domains.
$$
\boldsymbol {f} _ {i} ^ {\prime m} = \boldsymbol {\mu} ^ {m} + \epsilon \cdot \exp (\boldsymbol {a} ^ {m}), \quad \epsilon \sim \mathcal {N} (0, 1) \wedge - \alpha \leq \epsilon \leq \alpha , \tag {12}
$$
where  $\alpha$  is a truncation hyper-parameter to control the sampling area. Then discriminator  $\mathfrak{D}$  is trained to distinguish  $\{\pmb{f}_i^t\}_{i=1}^{N^t}$  from  $\{\pmb{f}_{1}^{\prime m}, \dots, \pmb{f}_{B}^{\prime m}\}_{m=1}^{t-1}$  and the feature extractor tries to confuse  $\mathfrak{D}$ . Such adversarial process is achieved by the following loss  $\mathcal{L}_{adv}^t$ :
$$
\max  _ {\mathcal {E}} \min  _ {\mathfrak {D}} \mathcal {L} _ {a d v} ^ {t} = \frac {1}{2} \left(\frac {1}{B \times (t - 1)} \sum_ {m = 1} ^ {t - 1} \sum_ {j = 1} ^ {B} - \log \left(\mathfrak {D} \left(\boldsymbol {f} _ {j} ^ {\prime m}\right)\right) + \frac {1}{N ^ {t}} \sum_ {j = 1} ^ {N ^ {t}} - \log \left(1 - \mathfrak {D} \left(\boldsymbol {f} _ {j} ^ {t}\right)\right)\right), \quad t \geq 2. \tag {13}
$$
In practice, the gradient reverse layer (GRL) [15] is used to achieve the adversarial training. To sum up, at time point  $t_t$ , the model is trained to minimize the following total loss  $\mathcal{L}_{total}^t$ :
$$
\mathcal {L} _ {\text {t o t a l}} ^ {t} = \mathcal {L} _ {c e} ^ {t} + \mathcal {L} _ {\text {s t a n}} ^ {t} + \sum_ {w = 1} ^ {W} \mathcal {L} _ {\text {c o n}} ^ {t, w} + \lambda \mathcal {L} _ {\text {a d v}} ^ {t}, \tag {14}
$$
Table 1: Accuracy (%) on Yearbook, RMNIST and fMoW. The best and second-best results in CDGTD setup are bolded and underlined. (Yearbook:  $K = 5$ , RMNIST:  $K = 3$ , fMoW:  $K = 3$ )
Method	Conference	Incremental training	Access multiple domains	Yearbook Accuracy (%) ↑	RMNIST Accuracy (%) ↑	fMoW Accuracy (%) ↑
DT+1	OOD avg.	OOD worst	DT+1	OOD avg.	OOD worst	DT+1	OOD avg.	OOD worst
Offline	-	X	✓	89.30	88.46	86.81	98.15	92.14	83.89	72.43	59.76	49.85
IRM [2]	arXiv'19	X	✓	97.09	94.52	92.58	95.10	85.05	72.52	64.77	54.92	46.51
CORAL [50]	ECCV Workshops'16	X	✓	95.94	91.79	88.84	93.04	79.10	62.96	62.14	51.42	42.19
Mixup [61]	ICLR'18	X	✓	94.98	91.12	88.35	97.11	89.66	79.63	70.27	57.73	48.04
LISA [57]	ICML'22	X	✓	95.51	92.97	91.29	96.21	87.04	75.15	70.05	55.52	44.61
CDOT [43]	arXiv'19	X	✓	95.17	92.90	91.46	97.96	90.19	79.67	-	-	-
CIDA [55]	ICML'20	X	✓	92.36	90.67	88.45	97.43	89.19	78.32	-	-	-
GI [41]	NeurIPS'21	X	✓	97.42	96.37	95.73	97.78	91.00	82.46	61.62	50.83	42.78
LSSAE [44]	ICML'22	X	✓	93.93	92.12	88.75	96.73	90.36	82.13	59.15	48.66	41.38
IncFinetune	-	✓	X	96.61	94.72	93.48	98.62	92.80	84.61	65.52	53.99	45.23
Mixup [61]	ICLR'18	✓	X	90.21	89.83	88.43	98.43	92.38	83.45	64.84	52.00	42.54
SimCLR [11]	ICML'20	✓	X	95.94	93.07	89.65	98.23	90.98	81.05	64.97	53.20	44.71
SwAV [8]	NeurIPS'20	✓	X	97.37	94.27	91.44	98.08	90.85	80.96	66.47	54.51	45.29
EWC [22]	arXiv'16	✓	X	97.18	95.12	93.64	98.56	92.02	82.80	66.23	54.55	45.80
SI [60]	ICML'17	✓	X	97.09	94.67	93.48	98.61	93.27	85.65	66.61	54.89	46.46
A-GEM [10]	ICLR'19	✓	X	94.36	90.96	88.88	95.99	86.95	75.45	54.54	47.61	41.13
SGP [47]	AAAI'23	✓	X	95.65	92.92	91.39	97.12	88.97	78.05	-	-	-
DRAIN [3]	ICLR'23	✓	X	96.23	94.71	93.73	98.52	93.09	85.75	67.22	55.05	46.24
EvoS	-	✓	X	97.37	95.53	94.78	98.64	93.84	87.04	67.18	54.64	45.86
For fMoW, backbone DenseNet-121 is too big to apply full GI and DRAIN. So we apply DRAIN only to the classifier and apply GI without the fine-tuning stage.
where  $\lambda$  is a hyper-parameter to balance the loss tradeoff. Once the training procedure at time point  $t_t$  is finished, we store the learned/generated statistics of current domain  $\mathcal{D}^t$  into memory pool  $\mathcal{M}$  by
$$
\boldsymbol {\mu} ^ {t}, \boldsymbol {a} ^ {t} \leftarrow \hat {\boldsymbol {\mu}} ^ {t}, \hat {\boldsymbol {a}} ^ {t}. \tag {15}
$$
In the inference stage, we use MSAM to generate future statistics  $\hat{s}^{T + k}$  based on the statistics  $\{s^t\}_{t = 1}^{T + k - 1}$  in memory pool  $\mathcal{M}$  and store it into  $\mathcal{M}$  as Eq. 15 for the generation at next time point. Due to space limitation, the training and testing procedures are provided in the appendix.
Thanks to the work in [56], several real-world datasets with distribution shifts over time have been available. And we evaluate EvoS on three image classification datasets (Yearbook and fMoW from [56] and RMNIST) and two text classification datasets (Huffpost and Arxiv from [56]). Yearbook collects data from 1930 to 2013, where we treat every four years as a domain and use the first 16 domains for training ( $T = 16$ ), the last 5 domains for testing ( $K = 5$ ). fMoW includes data of 16 years and we set  $T = 13$ ,  $K = 3$ . RMNIST contains 9 domains with  $T = 6$ ,  $K = 3$ . Huffpost includes data of 7 years with  $T = 4$ ,  $K = 3$ . Arxiv collects data of 172 categories for 16 years, with  $T = 9$ ,  $K = 7$ . For each training domain of all datasets, we randomly select  $90\%$  data as training split and  $10\%$  data as validation split. Following the backbones in [56], we use a 4-layer convolutional network [56] for Yearbook, DenseNet-121 [18] pretrained on ImageNet for fMoW, pretrained DistilBERT base model [48] for Huffpost and Arxiv, and the ConvNet in [44] for RMNIST. For each attention module  $\mathcal{A}_w$  in MSAM, its dimension of head  $d_h$  is set to 8 and its number of heads is set to  $w \cdot n_h$ . Specifically,  $n_h$  is set to 16 for Yearbook, 32 for RMNIST, 64 for fMoW and 128 for Huffpost and Arxiv. For optimization, we use the Adam optimizer with  $lr = 1e - 3$  for Yearbook and RMNIST,  $lr = 2e - 4$  for fMoW and  $lr = 2e - 5$  for Huffpost and Arxiv. The batch size is set to 64 for all datasets. As for hyper-parameters, we select them via grid search using the validation splits of training domains and finally use  $\alpha = 2.0$  for RMNIST,  $\alpha = 1.0$  for others,  $\lambda = 1.0$ ,  $W = 3$  for all datasets. For all tasks, we report the mean of 3 random trials. Due to space limitations, please refer to appendix for more details. Code is available at https://github.com/BIT-DA/EvoS.
In addition to the incremental training scenario, we also provide results in non-incremental scenario with all source domains simultaneously available, which serves as upper bounds. Among compared baselines, "Offline" denotes merging all source domains into one domain and training the model with the merged domain, while "IncFinetune" represents incrementally training the model in a domain-by-domain fashion. For each dataset, we report the accuracy on the nearest target domain  $(\mathcal{D}^{T + 1})$
Table 2: Accuracy (%) on Huffpost and Arxiv. The best and second-best results in CDGTD setup are bolded and underlined. (Huffpost:  $K = 3$ , Axriv:  $K = 7$ )
Method	Conference	Incremental training	Access multiple domains	Huffpost Accuracy (%) ↑	Arxiv Accuracy (%) ↑
DT+1	OOD avg.	OOD worst	DT+1	OOD avg.	OOD worst
Offline	-	X	✓	72.74	71.50	69.63	57.49	52.38	49.28
IRM [2]	arXiv'19	X	✓	71.04	70.31	68.97	51.11	45.89	42.86
CORAL [50]	ECCV Workshops'16	X	✓	71.34	70.08	68.68	50.98	45.77	42.71
Mixup [61]	ICLR'18	X	✓	73.34	71.16	69.29	57.58	52.77	49.62
LISA [57]	ICML'22	X	✓	72.19	70.24	68.60	56.53	52.41	49.67
GI [41]	NeurIPS'21	X	✓	68.06	66.32	64.64	53.43	49.19	46.13
IncFinetune	-	✓	X	73.57	71.98	69.80	56.22	52.43	49.37
Mixup [61]	ICLR'18	✓	X	73.07	71.52	69.44	56.64	52.95	49.97
EWC [22]	arXiv'16	✓	X	73.64	71.53	68.99	56.60	52.78	49.73
SI [60]	ICML'17	✓	X	72.58	71.50	69.61	49.98	47.27	44.77
A-GEM [10]	ICLR'19	✓	X	72.23	71.16	69.10	52.02	48.91	46.03
DRAIN [3]	ICLR'23	✓	X	73.42	71.75	69.69	56.04	52.07	48.97
EvoS	-	✓	X	73.42	72.36	70.19	56.60	53.15	50.19
For Huffpost and Arxiv, backbone DistilBERT-base is too big to apply the full GI and DRAIN. So we apply DRAIN only to the classifier and apply GI without the fine-tuning stage.
the average and worst accuracy of future  $K$  domains ("OOD avg.":  $\frac{1}{K}\sum_{k=1}^{K} Acc(\mathcal{D}^{T+k})$ , "OOD worst":  $\min_{k \in \{1,2,\dots,K\}} Acc(\mathcal{D}^{T+k})$ ).
Results on image classification tasks are provided in Table 5, where we can observe that EvoS consistently achieves superior performance across three reported metrics on Yearbook and RMNIST datasets, compared with the methods in incremental training scenario. Especially, EvoS outperforms DRAIN by over  $1\%$  according to the metric "OOD worst" on Yearbook and RMNIST, which means that our EvoS learns more robust evolving patterns. In addition, we notice that conventional DG methods IRM [2] and CORAL [50] perform worse than temporal DG method GI [41] on Yearbook and RMNIST, showing the importance of leveraging evolutionary patterns for generalization over temporal drifts. Though being effective for the small networks used in Yearbook and RMNIST, the advantage of GI disappears on fMoW dataset with backbone DenseNet-121, where its first-order Taylor expansion requires extremely huge computing resources (over 80G). And our available resources cannot afford it. When ablating the Taylor expansion, the performance of GI on fMoW is not the best. So GI is hard to expand on larger models and the similar issue also exists in DRAIN and LSSAE. Finally, we find that for dataset fMoW, not only our method performs unsatisfactorily but also other methods, except for the "offline" method (i.e., merging all source domain into one domain to train the model). We infer this may be due to the temporal drift is not well presented in this dataset.
Results on text classification tasks are shown in Table 6, where EvoS consistently achieves the best performances according to the average and worst accuracy of future  $K$  domains under the CDGTD setting. This can be owed to the more robust evolving patterns captured by our multi-scale attention module (MSAM). Similarly, IRM and CORAL show obvious performance drop compared with DRAIN, once again demonstrating the necessity to learn evolving patterns for the problem of CDGTD. And the large performance gap between continual learning methods (i.e., SI and A-GEM) and our method EvoS on dataset Arxiv verifies that our method EvoS fully utilizes the historical knowledge to learn evolutionary patterns, while SI and A-GEM do not consider this.
Ablation Study. Firstly, we study the influence of the stopping gradient  $\mathrm{sg}(\cdot)$  in the losses  $\mathcal{L}_{ce}^{t}$  and  $\mathcal{L}_{stan}^{t}$ . The results of variant A and B in Table 3 almost degenerate to random predictions,
compared with variant C. This suggests that using  $\mathrm{sg}(\cdot)$  in loss  $\mathcal{L}_{stan}^{t}$  is essential. Otherwise, the gradient of  $\mathcal{L}_{stan}^{t}$  would largely distort the learning of feature encoder, causing training collapse. Meanwhile, the inferior performance of variant C to EvoS also indicates that the gradient of  $\mathcal{L}_{ce}^{t}$  should not interfere with the learning of MSAM. Secondly, we ablate the losses  $\mathcal{L}_{adv}^{t}$  and  $\mathcal{L}_{con}^{t,w}$  to testify their necessity. We can see that the results of
Table 3: Ablation study of EvoS on dataset Yearbook.
Method	Scale	Loss	Using sg(·)?	Using truncation α?	ODQ avg.
single multi	Lctadv	Lctw, con	Lctce	Lctstan
Variant A	-	✓	✓	✓	No	No	Yes	51.28
Variant B	-	✓	✓	✓	Yes	No	Yes	55.37
Variant C	-	✓	✓	✓	No	Yes	Yes	93.97
Variant D	-	✓	-	-	Yes	Yes	Yes	93.25
Variant E	-	✓	✓	-	Yes	Yes	Yes	94.75
Variant F	-	✓	-	✓	Yes	Yes	Yes	94.27
Variant G	-	✓	✓	✓	Yes	Yes	No	94.46
EvoS	-	✓	✓	✓	Yes	Yes	Yes	95.53
Variant H	✓	-	✓	✓	Yes	Yes	Yes	94.09
Table 4: Results of using historical domain distributions in different ways during adversarial training.
Method	Used historical domain distribution in Ladv	Yearbook Accuracy (%) ↑	RMNST Accuracy (%) ↑	fMoW Accuracy (%) ↑	Huffpost Accuracy (%) ↑	Arxiv Accuracy (%) ↑
D T+1	OOD avg.	OOD worst	D T+1	OOD avg.	OOD worst	D T+1	OOD avg.	OOD worst	D T+1	OOD avg.	OOD worst	D T+1	OOD avg.	OOD worst
EvoS†	random one	96.56	95.40	94.46	98.05	93.04	85.46	66.62	53.81	44.76	72.99	71.57	69.14	55.89	52.45	49.52
EvoS	all historical	97.37	95.53	94.78	98.64	93.84	87.04	67.18	54.64	45.86	73.42	72.36	70.19	56.60	53.15	50.19
EvoS† denotes that we randomly select a domain distribution from the memory pool  $\mathcal{M}$  to sample a batch of  $B$  features for participating in the adversarial training.
(a) Discrepancy between real and generated statistics.
Figure 2: (a): The  $l - 2$  norm of the difference  $\Delta$  between the statistics calculated on each test domain and the statistics generated by MSAM on Yearbook. "Ours" refers to the way in Eq. 10 and "Compared" refers to minimizing the  $l - 2$  norm of  $\Delta$  between statistics calculated in each batch and that generated by MSAM. (b): Effects of using different numbers of heads and scales on Yearbook.
(b) Effect of the number of heads and scale.
variant D, E and F are all inferior to EvoS. Such results verify two points. One is that the adversarial training conduces to better generalization and mitigating the problem of overfitting. And the other is that the consistency between generated statistic tokens in the window and real statistic tokens in the memory pool imposes stronger constraint to attention module  $\mathcal{A}_w$ , helping to model the evolving pattern more accurately. Thirdly, we testify the influence of truncation when sampling features from normal distributions. The worse result of variant G than EvoS may be due to the sampled outliers hinder the learning of a shared feature space. Finally, the results of variant H and EvoS compare single-scale and multi-scale attention. The better result of EvoS shows the superiority of MSAM.
Last but not least, in Table 4, we investigate the effect of randomly selecting a historical domain distribution vs using all historical domain distributions in the adversarial training. From the results, we see that randomly selecting one historical domain distribution performs worse. This may be because the feature space to be aligned frequently changes if using this manner, making the optimization challenging. By contrast, it is more stable to simultaneously leverage all the preserved domain distributions in the memory pool  $\mathcal{M}$  in each iteration for the adversarial training.
Choices of  $\mathcal{L}_{stan}^t$ . In addition to training  $\mathcal{A}_w$  via loss  $\mathcal{L}_{stan}^t$  in Eq. 10, we also try another way. Concretely, we directly minimize the  $l-2$  norm of the difference between statistics calculated in each batch and that generated by MSAM, i.e.,  $\mathcal{L}_{stan}^t = \left\| \frac{1}{B}\sum_{i=1}^{B}\mathrm{sg}(\pmb{f}_i^t) - \hat{\pmb{\mu}}^t\right\|_2 + \left\| \frac{1}{B-1}\sum_{i=1}^{B}(sg(\pmb{f}_i^t) - \frac{1}{B}\sum_{j=1}^{B}\mathrm{sg}(\pmb{f}_j^t))^2 - (\hat{\pmb{\sigma}}^t)^2\right\|_2$ . Fig. 2(a) plots the discrepancy between generated and real statistics on each test domain of Yearbook when using the two ways to train the attention module. We see that the way used in Sec. 3.3 yields a smaller discrepancy between generated and real statistics, implying that the learned evolving pattern is more accurate. So we choose to use it throughout the experiments.
Effect of Number of Heads and Scale. We further investigate the effect of number of heads and scale of MSAM in Fig. 2(b). Varying the number of heads, the performance exhibits an inverted V-shaped trend, because too few heads would hamper the learning capability of the attention module while too many heads could lead to overfitting. As for difference scales (i.e., the maximum length of sliding time window), we find that multi-scales  $(W \geq 1)$  yields better performance than single-scale  $(W = 1)$ , showing the superiority of MSAM. Yet, more scales do not always produce better results. One possible explanation is that an excessively large scale relative to the domain sequence length leads to an inadequate number of sliding windows, hampering the learning of attention module.
Visualization of Decision Boundary. In this qualitative experiment, we visualize the change of decision boundary on the inter-twinning moons 2D problem in a gradual manner. Concretely, the 2-Moons dataset in [41] is used, where 10 domains (0 to 9) are obtained by rotating data points counter-clockwise in units of  $18^{\circ}$ . In Figure 3, the model is sequentially trained using EvoS until the  $t$ -th domain is finished, and then we visualize the decision boundary on current domain  $\mathcal{D}^t$  and the next future domain  $\mathcal{D}^{t + 1}$ . From the results, we can observe that the decision boundary successfully adapts to future domains, showing that EvoS can truly capture the underlying temporal drift of data.
Figure 3: Visualization of decision boundaries for the model at current time point  $\mathfrak{t}_t$  on inter-twinning moons 2D problem. Blue line and points are the decision boundary and samples in current domain  $\mathcal{D}^t$ , and red line and points are the decision boundary and samples in the next future domain  $\mathcal{D}^{t + 1}$ .
(a) t-SNE visualization of test domains.
(b) Effect of pool length  $L$
Figure 4: (a): Visualization of standardized features from each test domain on Yearbook and RMNIST. Different colors denote different domains. (b): Effect of the memory pool length  $L$  on dataset Yearbook. (c):Hyper-parameter sensitivity of  $\alpha$  and  $\lambda$  on Yearbook
(c) Sensitivity of  $\alpha, \lambda$ .
Effect of Memory Pool Length. In the main experiments, we do not restrict the size of the memory pool  $\mathcal{M}$ , since the datasets used in our paper has a moderate number of domains and the memory cost is small. Nevertheless, a fixed memory pool size is more practical when considering a lifelong process, i.e.,  $T \to \infty$ . Thus, we additionally conduct the experiment where the memory pool  $\mathcal{M}$  is implemented as a FIFO queue with different fixed length  $L$ . That is, only the statistics for up to the  $L$  most recent historical domains can be stored. The results on Yearbook in Fig. 4(b) demonstrate that our method generally performs well and a relatively large memory pool length is better.
t-SNE Visualization of Standardized Features. To verify our method appropriate for generalization, we visualize the standardized features of all target domains via t-SNE [38] on RMNIST and Yearbook in Fig. 4(a). It can be observed that the standardized features of target domains are well aligned, suggesting that MASM can capture evolving patterns effectively and feature standardization helps address temporal drift appropriately. These enable EvoS to achieve generalization on future domains.
Hyper-parameter Sensitivity.  $\alpha$  and  $\lambda$  control the truncation range and the tradeoff the adversarial loss  $\mathcal{L}_{adv}$ , respectively. Fig. 4(c) shows the sensitivity of EvoS to them on Yearbook, where  $\alpha \in \{0.1, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0\}$  and  $\lambda \in \{0.1, 0.5, 1.0, 1.5, 2.0, 5.0, 10.0, 15.0, 20.0\}$ . We see that EvoS is more sensitive to  $\lambda$  than  $\alpha$ , and large  $\lambda$  values significantly worsen performance. In practice, we suggest selecting  $\lambda$  from a range  $\leq 1$ . For  $\alpha$ , its effect is modest and  $\alpha = 1$  generally works well.