SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Memory Complexity of  $\mathcal{M}$ . Assuming that there are  $T$  historical domains and the dimension of statistic vectors is  $d_f$ , then the memory complexity of the memory pool  $\mathcal{M}$  is  $\mathcal{O}(T \cdot d_f)$ . In practice, after being processed by deep neural networks, the dimension of pooled features is usually much smaller than that of original inputs. Moreover, only two vectors need to be stored per domain. Hence, the memory cost of  $\mathcal{M}$  is relatively small, compared with sample replay-based CL methods. Concretely, Table 7 provides the memory cost of  $\mathcal{M}$  and the increment of GPU memory cost for EvoS on Yearbook, RMNIST and fMoW datasets with batch size 64.
Table 7: Memory cost on Yearbook, RMNIST and fMoW datasets.
GPU memory cost (GB) of different methods
Method	Dataset
Yearbook	RMNIST	fMoW
Backbone
4-layer CNN [56]	ConvNet [44]	DenseNet-121 [18]
IncFinetune	1.72	1.84	10.69
EvoS	1.94	2.09	11.04
Increment Δ (GB)	0.22	0.25	0.35
Memory cost (MB) of the memory pool M
EvoS	5.25	9.00	32.00
Table 8: Model parameters (MB) of different methods on Yearbook, RMNIST and fMoW.
Method	Dataset
Yearbook	RMNIST	fMoW
Backbone
4-layer CNN [56]	ConvNet [44]	DenseNet-121 [18]
LSSAE [44]	4.70	23.25	90.92
DRAIN [3]	7.51	184.29	1113.85
EvoS	1.94	15.81	56.11
Model Complexity. Here, we measure the model complexity by the number of parameters. Specifically, Table 8 presents the model complexity of our method EvoS and other two temporal DG methods LSSAE [44] and DRAIN [3] on Yearbook, RMNIST and fMoW datasets. LSSAE [44] introduces sequential autoencoder to explore the underlying continuous structure in the latent space of deep neural networks, where the complicated VAE and LSTM networks require lots of parameters. And DRAIN [3] needs to encode and decode the entire network parameters, which also requires a great number of parameters for large backbone networks. By contrast, our EvoS is overall less complex than these temporal DG methods, and is more friendly to the relatively large backbone network.
Results under Eval-Stream Manner. In this part, we additionally provide the results when adopting the evaluation manner of Eval-Stream in [56]. Specifically, Eval-Stream denotes the evaluation with domain stream, i.e., the model is evaluated at each timestamp using the average and worst performance on the next  $K$  timestamps. Formally, given a sequence of domains with total length  $\mathcal{T}$ , the average performance  $Avg_{stream}$  and worst performance  $Worst_{stream}$  under the strategy of Eval-Stream are defined as  $Avg_{stream} = \frac{1}{\mathcal{T} - K}\sum_{i=1}^{\mathcal{T} - K}\frac{1}{K}\sum_{j=i+1}^{i + K}Acc_i(\mathcal{D}^j)$ ,  $Worst_{stream} = \frac{1}{\mathcal{T} - K}\sum_{i=1}^{\mathcal{T} - K}\min_{j\in\{i+1,\cdots,i+K\}}Acc_i(\mathcal{D}^j)$ , where  $Acc_i(\mathcal{D}^j)$  is the accuracy on domain  $\mathcal{D}^j$  when using the model at the  $i$ -th timestamp. And Table 9 shows the results on Yearbook and Huffpost datasets when using the Eval-Stream evaluation manner. According to the results, we see that EvoS still outperforms other baselines, showing that our method is better at handling the problem of continual domain generalization over temporal drift.
Table 9: Accuracy (%) on Yearbook and Huffpost under the evaluation manner of Eval-Stream. (Yearbook:  $\mathcal{T} = 21$ ,  $K = 5$ , Huffpost:  $\mathcal{T} = 7$ ,  $K = 3$ )
Method	Conference	Yearbook Accuracy (%) ↑	Huffpost Accuracy (%) ↑
\(Avg_{stream}\)	\(Worst_{stream}\)	\(Avg_{stream}\)	\(Worst_{stream}\)
IncFinetune	-	89.67	83.56	67.97	64.11
Mixup [61]	ICLR'18	84.79	78.69	67.33	63.48
SimCLR [11]	ICML'20	89.50	83.16	-	-
SwAV [8]	NeurIPS'20	90.05	84.08	-	-
EWC [22]	arXiv'16	90.15	83.75	68.31	64.58
SI [60]	ICML'17	90.14	84.07	67.93	64.05
A-GEM [10]	ICLR'19	84.40	77.59	65.83	62.20
DRAIN [3]	ICLR'23	87.26	81.95	68.04	64.14
EvoS	-	90.43	84.48	68.66	64.97
Table 10: Misclassification error (%) on 2-Moons, ONP and Elec2.  $\left( {K = 1}\right)$
Method	Conference	Misclassification error (in %) ↓
2-Moons	ONP	Elec2
Offline	-	22.4±4.6	33.8±0.6	23.0±3.1
LastDomain	-	14.9±0.9	36.0±0.2	25.8±0.6
IncFinetune	-	16.7±3.4	34.0±0.3	27.3±4.2
CDOT [43]	arXiv'19	9.3±1.0	34.1±0.0	17.8±0.6
CIDA [55]	ICML'20	10.8±1.6	34.7±0.6	14.1±0.2
GI [41]	NeurIPS'21	3.5±1.4	36.4±0.8	16.9±0.7
DRAIN [3]	ICLR'23	3.2±1.2	38.3±1.2	12.7±0.8
EvoS	-	2.5±1.0	33.1±0.6	11.6±0.7
Results on More Datasets. In addition to the datasets provided in [56], we also run our method on the 2-Moons, ONP and Elec2 datasets used in [3]. Specifically, 2-Moons is a variant of the 2-entangled moons dataset by rotating data counter-clockwise in units of  $18^{\circ}$  to construct 10 domains, where the rotation angle is used to simulate the temporal shift. For 2-Moons, we use Adam optimizer and a MLP with two hidden layers of hidden size 64 and 128, and  $B = 64$ ,  $lr = 1e - 3$ ,  $\alpha = 1.0$ ,  $\lambda = 1.0$ ,  $W = 3$ ,  $d_h = 8$ ,  $n_h = 32$ ,  $T = 9$ ,  $K = 1$ . Online News Popularity (ONP) summarizes a heterogeneous set of features about articles published by Mashable in a period of two years. The dataset is split into 6 domains by time and the goal is to predict the number of shares in social networks (popularity). For ONP, we use Adam optimizer and a MLP with one hidden layer of hidden size 128, and  $B = 64$ ,  $lr = 1e - 4$ ,  $\alpha = 1.0$ ,  $\lambda = 1.0$ ,  $W = 3$ ,  $d_h = 8$ ,  $n_h = 32$ ,  $T = 5$ ,  $K = 1$ . Electrical Demand (Elec2) contains information about the demand of electricity in a particular province. Following [3, 41], the first 30 domains in Elec2 are used (two weeks as one time domain) and the task is to predict whether the demand of electricity in each period (of 30 mins) is higher or lower than the average demand over the last day. For Elec2, we use Adam optimizer and a MLP with two hidden layers of hidden size 128 and 128, and  $B = 64$ ,  $lr = 1e - 4$ ,  $\alpha = 1.0$ ,  $\lambda = 1.0$ ,  $W = 3$ ,  $d_h = 8$ ,  $n_h = 32$ ,  $T = 29$ ,  $K = 1$ . For these datasets, as in [3, 41], we use the last domain for testing and the rest for training. Please refer to [3] for more dataset details. The experimental results are given in Table 10, where the misclassification errors of compared baselines are all reported from DRAIN [3]. From Table 10, we can observe that our method EvoS still surpasses the most recent method DRAIN, affirming its effectiveness in temporal domain generalization.
Table 11: Data Subset Size for the Yearbook Dataset.
Domain	Interval	Training Split	Validation Split	All
1	1930 - 1933	758	87	845
2	1934 - 1937	1149	130	1279
3	1938 - 1941	949	108	1057
4	1942 - 1945	2353	263	2616
5	1946 - 1949	1229	138	1367
6	1950 - 1953	1082	122	1204
7	1954 - 1957	1646	185	1831
8	1958 - 1961	1295	146	1441
9	1962 - 1965	1468	166	1634
10	1966 - 1969	2227	249	2476
11	1970 - 1973	1634	183	1817
12	1974 - 1977	2238	250	2488
13	1978 - 1981	1553	175	1728
14	1982 - 1985	2331	261	2592
15	1986 - 1989	1792	201	1993
16	1990 - 1993	1729	195	1924
17	1994 - 1997	1882	211	2093
18	1998 - 2001	2136	239	2375
19	2002 - 2005	1868	210	2078
20	2006 - 2009	1010	114	1124
21	2010 - 2013	1102	125	1227
total	1930 - 2013	33431	3758	37189
Table 12: Data Subset Size for the fMoW Dataset.
Domain	Year	Training Split	Validation Split	All
1	2002	1676	227	1903
2	2003	2279	276	2555
3	2004	1755	240	1995
4	2005	2512	324	2836
5	2006	3155	406	3561
6	2007	1497	190	1687
7	2008	2261	298	2559
8	2009	7439	935	8374
9	2010	18957	2456	21413
10	2011	22111	2837	24948
11	2012	24704	3138	27842
12	2013	3465	385	3850
13	2014	5572	620	6192
14	2015	8885	988	9873
15	2016	14363	1596	15959
16	2017	5534	615	6149
total	2002-2017	126165	15531	141696
Table 13: Data Subset Size for the Huffpost Dataset.
Domain	Year	Training Split	Validation Split	All
1	2012	6701	744	7446
2	2013	7492	832	8325
3	2014	9539	1059	10599
4	2015	11826	1313	13140
5	2016	10548	1172	11721
6	2017	7907	878	8786
7	2018	3501	388	3890
total	2012-2018	57514	6386	63907
Table 14: Data Subset Size for the Arxiv Dataset.
Domain	Year	Training Split	Validation Split	All
1	2007	131550	14616	146167
2	2008	62460	6939	69400
3	2009	206244	22916	229161
4	2010	50665	5629	56295
5	2011	55741	6193	61935
6	2012	51678	5741	57420
7	2013	64951	7216	72168
8	2014	79498	8833	88332
9	2015	193979	21553	215533
10	2016	120682	13409	134092
11	2017	111024	12336	123361
12	2018	123891	13765	137657
13	2019	142767	15862	158630
14	2020	166014	18445	184460
15	2021	201241	22360	223602
16	2022	89765	9973	99739
total	2007-2022	1852150	205786	2057952