SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 8 presents the results of different methods on Seq-CIFAR100 with varying numbers of tasks. Here, the memory size is set as 500 for all the methods. As we can see, when the number of tasks varies, LODE still gives improvements on different methods. Furthermore, LODE (ESMER) still gets the best performance among all the methods.
Table 9 gives the results of additional methods that directly sample samples from  $\mathcal{D}_t\cup \mathcal{M}$ . Some methods like BIC and iCaRL, which directly sample samples from  $\mathcal{D}_t\cup \mathcal{M}$ , have been compared with our methods in the main text. Since all these methods keep an extra model in memory for distillation, we compare them with LODE (ESMER). As we can see, our method still gets the best performance.
Number of Tasks 1 (a) Seq-CIFAR10
Number of Tasks (b)Seq-CIFAR100
Loss Value (c) Seq-CIFAR10
(d) Seq-CIFAR100
Figure 7: (a) and (b) show the variation of the first task's accuracy on different datasets. (c) and (d) show the distribution of different losses over different datasets before learning the second task. Here, the frequency corresponding to each loss value represents the number of samples with this loss value.
C.5 More Analysis by Decoupling Loss
We also use dark experience replay++ (DER++) [7] to evaluate the impact of  $\ell_{ce}(\cdot; \mathcal{C}_n)$  and  $\ell_n(\cdot)$ . Figure 7 gives the results. The results are similar to that in ER. Specifically, removing  $\ell_n(\cdot)$  results in less forgetting of the first task than removing  $\ell_{ce}(\cdot; \mathcal{C}_n)$ . In other words,  $\ell_n(\cdot)$  leads to more forgetting than  $\ell_{ce}(\cdot; \mathcal{C}_n)$ . Furthermore, the value of  $\ell_n(\cdot)$  is larger than the value of  $\ell_{ce}(\cdot; \mathcal{C}_n)$  on both Seq-CIFAR10 and Seq-CIFAR100.
Table 8: Results on Seq-CIFAR100 with different numbers of tasks. The memory size is set to 500.
Keeping
Extra Model	T=4	T=10
no	PCR [25]	30.96±0.56	19.88±2.29
MIR [3]	30.44±0.65	22.01±0.68
ER-ACE [8]	45.71±0.59	36.95±0.86
ER [9]	30.72±0.27	20.45±1.38
LODE (ER)	45.60±0.24	37.32±0.74
DER++ [7]	43.68±0.86	36.30±0.85
LODE (DER++)	48.95±0.77	38.61±1.23
yes	CLS-ER [5]	54.82±1.26	43.05±0.64
TAMiL [6]	53.42±0.29	42.52±1.31
iCaRL [33]	50.97±1.04	42.25±0.62
BIC [43]	40.75±0.77	20.71±0.65
SSL [1]	39.39±0.36	33.06±0.86
ESMER [36]	55.97±1.22	45.54±0.87
LODE (ESMER)	57.92±0.30	46.37±0.39
Table 9: Results of more methods on Seq-CIFAR100 and Seq-TinyImageNet with memory size 500.
Seq-CIFAR100	Seq-TinyImageNet
iCaRL [33]	49.59±0.95	20.01±0.50
BIC [43]	37.06±0.60	29.82±0.88
SSIL [1]	41.61±0.37	16.80±0.71
PODNet [14]	45.95±0.45	26.10±1.71
UCIR [17]	47.56±0.43	26.22±0.84
LODE (ESMER)	55.06±0.35	32.15±0.17