SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Here, two hyper-parameters,  $\lambda_{p}$  and  $\lambda_{s}$ , balance the influence of each individual loss on the encoder's embedding distribution. In practice, the final performance of UDIL is not significantly affected by the values of  $\lambda_{p}$  and  $\lambda_{s}$ , and a wide range of values for these parameters can yield reasonable results.
C.4 Evaluation Metrics
In continual learning, many evaluation metrics are based on the Accuracy Matrix  $\pmb{R} \in \mathbb{R}^{T \times T}$ , where  $T$  represents the total number of tasks (domains). In the accuracy matrix  $\pmb{R}$ , the entry  $R_{i,j}$  corresponds to the accuracy of the model when evaluated on task  $j$  after training on task  $i$ . With this definition in mind, we primarily focus on the following specific metrics:
Average Accuracy (Avg. Acc.) up until domain  $t$  represents the average accuracy of the first  $t$  domains after training on these domains. We denote it as  $A_{t}$  and define it as follows:
$$
A _ {t} \triangleq \frac {1}{t} \sum_ {i = 1} ^ {t} R _ {t, i}. \tag {67}
$$
In most of the continual learning literature, the final average accuracy  $A_{T}$  is usually reported. In our paper, this metric is reported in the column labeled "overall". The average accuracy of a model is a crucial metric as it directly corresponds to the primary optimization goal of minimizing the error on all domains, as defined in Eqn. 3.
Additionally, to better illustrate the learning (and forgetting) process of a model across multiple domains, we propose the use of the "Avg. of Avg. Acc." metric  $A_{t_1:t_2}$ , which represents the average of average accuracies for a consecutive range of domains starting from domain  $t_1$  and ending at domain  $t_2$ . Specifically, we define this metric as follows:
$$
A _ {t _ {1}: t _ {2}} \triangleq \frac {1}{t _ {2} - t _ {1} + 1} \sum_ {i = t _ {1}} ^ {t _ {2}} A _ {i}. \tag {68}
$$
This metric provides a condensed representation of the trend in accuracy variation compared to directly displaying the entire series of average accuracies  $\{A_1, A_2, \dots, A_T\}$ . We report this Avg. of Avg. Acc. metric in all tables (except in Table 2 due to the limit of space).
Average Forgetting (i.e., 'Forgetting' in the main paper) defines the average of the largest drop of accuracy for each domain up till domain  $t$ . We denote this metric as  $F_{t}$  and define it as follows:
$$
F _ {t} \triangleq \frac {1}{t - 1} \sum_ {j = 1} ^ {t - 1} f _ {t} (j), \tag {69}
$$
where  $f_{t}(j)$  is the forgetting on domain  $j$  after the model completes the training on domain  $t$ , which is defined as:
$$
f _ {t} (j) \triangleq \max  _ {l \in [ t - 1 ]} \left\{R _ {l, j} - R _ {t, j} \right\}. \tag {70}
$$
Typically, the average forgetting is reported after training on the last domain  $T$ . Measuring forgetting is of great practical significance, especially when two models have similar average accuracies. It indicates how a model balances stability and plasticity. If a model  $\mathcal{P}$  achieves a reasonable final average accuracy across different domains but exhibits high forgetting, we can conclude that this model has high plasticity and low stability. It quickly adapts to new domains but at the expense of performance on past domains. On the other hand, if another model  $S$  has a similar average accuracy to  $\mathcal{P}$  but significantly lower average forgetting, we can infer that the model  $S$  has high stability and low plasticity. It sacrifices performance on recent domains to maintain a reasonable performance on past domains. Hence, to gain a comprehensive understanding of model performance, we focus on evaluating two key metrics: Avg. Acc. and Forgetting. These metrics provide insights into how models balance stability and plasticity and allow us to assess their overall performance across different domains.
Forward Transfer  $W_{t}$  quantifies the extent to which learning from past  $t - 1$  domains contributes to the performance on the next domain  $t$ . It is defined as follows:
$$
W _ {t} \triangleq \frac {1}{t - 1} \sum_ {i = 2} ^ {t} R _ {i - 1, i} - r _ {i}, \tag {71}
$$
where  $r_i$  is the accuracy of a randomly initialized model evaluated on domain  $i$ . For domain incremental learning, where the model does not have access to future domain data and does not explicitly optimize for higher Forward Transfer, the results of this metric are typically random. Therefore, we do not report this metric in the complete tables presented in this section.
C.5 Introduction to Baselines
We compare UDIL with the state-of-the-art continual learning methods that are either specifically designed for domain incremental learning or can be easily adapted to the domain incremental learning setting. Exemplar-free baselines include online Elastic Weight Consolidation (oEWC) [81], Synaptic Intelligence (SI) [105], and Learning without Forgetting (LwF) [52]. Memory-based domain incremental learning baselines include Gradient Episodic Memory (GEM) [58], Averaged Gradient Episodic Memory (A-Geom) [10], Experience Replay (ER) [75], Dark Experience Replay  $(\mathbf{DER} + + )$  [8], and two recent methods, Complementary Learning System based Experience Replay (CLS-ER) [4] and Error Sensitivity Modulation based Experience Replay (ESM-ER) [80]. In addition, we implement the fine-tuning (Fine-tune) [52] and joint-training (Joint) as the performance lower bound and upper bound (Oracle). Here we provide a short description of the primary idea of the memory-based domain incremental learning baselines.
- GEM [58]: The baseline method that uses the memory to provide additional optimization constraints during learning the current domain. Specifically, the update of the model cannot point towards the direction at which the loss of any exemplar increases.
- A-GEM [10]: The improved baseline method where the constraints of GEM are averaged as one, which shortens the computational time significantly.
- ER [75]: The fundamental memory-based domain incremental learning framework where the mini-batch of the memory is regularly replayed with the current domain data.
- DER++ [8]: A simple yet effective replay-based method where an additional logits distillation (dubbed "dark experience replay") is applied compared to the vanilla ER.
- CLS-ER [4]: A complementary learning system inspired replay method, where two exponential moving average models are used to serve as the semantic memory, which provides the logits distillation target during training.
- ESM-ER [80]: An improved version of CLS-ER, where the effect of large errors when learning the current domain is reduced, dubbed "error sensitivity modulation".
C.6 Training Schemes
Training Process. For each group of experiments, we run three rounds with different seeds and report the mean and standard deviation of the results. We follow the optimal configurations (epochs and learning rate) stated in [8, 80] for the baselines in  $P$ -MNIST and  $R$ -MNIST dataset. For HD-Balls and Seq-CORE50, we first search for the optimal training configuration for the joint learning, and then grid-search the configuration in a small range near it for the baselines listed above. For our UDIL framework, as it involves adversarial training for the domain embedding alignment, we typically need a configuration that has larger number of epochs and smaller learning rate. We use a simple grid search to achieve the optimal configuration for it as well.
Model Architectures. For the baseline methods and UDIL in the same dataset, we adopt the same backbone neural architectures to ensure fair comparison. In HD-Balls, we adopt the same multi-layer perceptron with the same separation of encoder and decoder as in CIDA [94], where the hidden dimension is set to 800. In  $P$ -MNIST and  $R$ -MNIST, we adopt the same multi-layer perceptron architecture as in DER++ [8] with hidden dimension set to 800 as well. In Seq-CORE50, we use the ResNet18 [33] as our backbone architecture for all the methods, where the layers before the final average pooling are treated as the encoder  $e$ , and the remaining part is treated as the predictor  $p$ .
(a) Joint (91.083%, Oracle)
(b) GEM (76.619%)
(c) A-Geom (59.654%)
(d) ER (82.255%)
(e) DER++ (79.322%)
Figure 3: More Results on HD-Balls. Data is colored according to domain ID. All data is plotted after PCA [6]. (a-h) Accuracy and embeddings learned by Joint (oracle), UDIL, and six baselines with memory. Joint, as the oracle, naturally aligns different domains, and UDIL outperforms all baselines in terms of embedding alignment and accuracy.
(f) CLS-ER (85.844%)
(g) ESM-ER (82.192%)
(h)UDIL  $(86.872\%)$  Ours
Hyperparameter Setting. For setting the hyper-parameter embedding alignment strength coefficient  $\lambda_{d}$  and parameter  $C$  that models the combined effect of VC-dimension  $d$  and error tolerance  $\delta$ , we use grid search for each dataset, where the range  $\lambda_{d} \in [0.01, 100]$  and  $C \in [0, 1000]$  are used.
D Additional Empirical Results
This section presents additional empirical results of the UDIL algorithm. Sec. D.1 will show the additional results on different constraints with varying memory sizes. Sec. D.2 provides additional qualitative results: visualization of embedding distributions, to showcase the importance of the embedding alignment across domains.
D.1 Empirical Results on Varying Memory Sizes
Here we present additional empirical results to validate the effectiveness of our UDIL framework using varying memory sizes. The evaluation is conducted on three real-world datasets, as shown in Table 5, Table 6, and Table 7. By increasing the memory size from 400 to 800 in Table 5 and 6 and from 500 to 1000 in Table 7, we can investigate the impact of having access to a larger pool of past experiences on the continual learning performance, which might occur when the constraint on memory capacity is relaxed. This allows us to study the benefits of a more extensive memory in terms of knowledge retention and performance improvement. On the other hand, by further decreasing the memory size to the extreme of 200 in Table 5 and 6, we can explore the consequences of severely limited memory capacity. This scenario simulates situations where memory constraints are extremely tight, and the model can only retain a small fraction of past domain data, for example, a model deployed on edge devices. To ensure a fair comparison, here we use the same best configuration found in the main body of this work.
The results in all three tables demonstrate a clear advantage of our UDIL framework when the memory size is limited. In  $P$ -MNIST and  $R$ -MNIST, when the memory size  $|\mathcal{M}| = 200$ , the overall performance of UDIL reaches  $91.483\%$  and  $82.796\%$  respectively, which outperforms the second best model  $\mathrm{DER}++$  by  $0.757\%$  and a remarkable  $6.125\%$ . In Seq-CORE50, when the memory size  $|\mathcal{M}| = 500$  is set, UDIL holds a  $3.474\%$  lead compared to the second best result. When the memory size is larger, the gap between UDIL and the baseline models is smaller. This is because when the memory constraint is relaxed, all the continual learning models should be at least closer to the performance upper bound, i.e., joint learning or 'Joint (Oracle)' in the tables, causing the
indistinguishable results among each other. Apparently, DER++ favors larger memory more than UDIL, while UDIL can still maintain a narrow lead in the large scale dataset Seq-CORe50.
D.2 Visualization of Embedding Spaces
Here we provide more embedding space visualization results for the baselines with the utilization of memory, shown in Fig. 3. As one of the primary objectives of our algorithm, embedding space alignment across multiple domains naturally follows the pattern shown in the joint learning and therefore leads to a higher performance.
Table 5: Performances  $(\%)$  evaluated on  $P$  -MNIST. Average Accuracy (Avg. Acc.) and Forgetting are reported to measure the methods' performance. "↑" and "↓" mean higher and lower numbers are better, respectively. We use boldface and underlining to denote the best and the second-best performance, respectively. We use "-" to denote "not applicable".
Method	Buffer	D1:5	D6:10	D11:15	D16:20	Overall
Avg. Acc (↑)	Avg. Acc (↑)	Forgetting (↓)
Fine-tune	-	92.506±2.062	87.088±1.337	81.295±2.372	72.807±1.817	70.102±2.945	27.522±3.042
oEWC [81]	-	92.415±0.816	87.988±1.607	83.098±1.843	78.670±0.902	78.476±1.223	18.068±1.321
SI [105]	-	92.282±0.862	87.986±1.622	83.698±1.220	79.669±0.709	79.045±1.357	17.409±1.446
LwF [52]	-	95.025±0.487	91.402±1.546	83.984±2.103	76.046±2.004	73.545±2.646	24.556±2.789
GEM [58]	93.310±0.374	91.900±0.456	89.813±0.914	87.251±0.524	86.729±0.203	9.430±0.156
A-GEM [10]	93.326±0.363	91.466±0.605	89.048±1.005	86.518±0.604	85.712±0.228	10.485±0.196
ER [75]	94.087±0.762	92.397±0.464	89.999±1.060	87.492±0.448	86.963±0.303	9.273±0.255
DER++ [8]	200	94.708±0.451	94.582±0.158	93.271±0.585	90.980±0.610	90.333±0.587	6.110±0.545
CLS-ER [4]	94.761±0.340	93.943±0.197	92.725±0.566	91.150±0.357	90.726±0.218	5.428±0.252
ESM-ER [80]	95.198±0.236	94.029±0.427	91.710±1.056	88.181±1.021	86.851±0.858	10.007±0.864
UDIL (Ours)	95.747±0.486	94.695±0.256	93.756±0.343	92.254±0.564	91.483±0.270	4.399±0.314
GEM [58]	93.557±0.225	92.635±0.306	91.246±0.492	89.565±0.342	89.097±0.149	6.975±0.167
A-GEM [10]	93.432±0.333	92.064±0.439	90.038±0.726	87.988±0.335	87.560±0.087	8.577±0.053
ER [75]	93.525±1.101	91.649±0.362	90.426±0.456	88.728±0.353	88.339±0.044	7.180±0.029
DER++ [8]	400	94.952±0.403	95.089±0.075	94.458±0.328	93.257±0.249	92.950±0.361	3.378±0.245
CLS-ER [4]	94.262±0.649	93.195±0.148	92.623±0.195	91.839±0.187	91.598±0.117	3.795±0.144
ESM-ER [80]	95.413±0.139	94.654±0.314	93.353±0.588	91.022±0.781	89.829±0.698	6.888±0.738
UDIL (Ours)	95.992±0.349	95.026±0.250	94.212±0.280	93.094±0.326	92.666±0.108	2.853±0.107
GEM [58]	93.717±0.177	93.116±0.206	92.166±0.335	91.076±0.342	90.609±0.364	5.393±0.417
A-GEM [10]	93.612±0.241	92.523±0.375	90.718±0.739	88.543±0.391	88.020±0.851	8.081±0.867
ER [75]	93.827±0.871	92.457±0.217	91.688±0.277	90.617±0.289	90.252±0.056	5.188±0.045
DER++ [8]	800	95.295±0.317	95.539±0.041	95.099±0.187	94.423±0.151	94.227±0.261	2.106±0.161
CLS-ER [4]	94.463±0.537	93.567±0.093	93.182±0.137	92.744±0.112	92.578±0.152	2.803±0.183
ESM-ER [80]	95.567±0.150	95.136±0.202	94.301±0.347	92.981±0.397	92.408±0.387	4.170±0.357
UDIL (Ours)	96.082±0.313	95.207±0.196	94.642±0.156	93.997±0.194	93.724±0.043	1.633±0.035
Joint (Oracle)∞	-	-	-	-	96.368±0.042	-
Table 6: Performances (%) evaluated on R-MNIST. Average Accuracy (Avg. Acc.) and Forgetting are reported to measure the methods' performance. “↑” and “↓” mean higher and lower numbers are better, respectively. We use boldface and underlining to denote the best and the second-best performance, respectively. We use “-” to denote “not appliable”.
Method	Buffer	D1:5	D6:10	D11:15	D16:20	Overall
Avg. Acc (↑)	Avg. Acc (↑)	Forgetting (↓)
Fine-tune	-	92.961±2.683	76.617±8.011	60.212±3.688	49.793±1.552	47.803±1.703	52.281±1.797
oEWC [81]	-	91.765±2.286	76.226±7.622	60.320±3.892	50.505±1.772	48.203±0.827	51.181±0.867
SI [105]	-	91.867±2.272	76.801±7.391	60.956±3.504	50.301±1.538	48.251±1.381	51.053±1.507
LwF [52]	-	95.174±1.154	83.044±5.935	65.899±4.061	55.980±1.296	54.709±0.515	45.473±0.565
GEM [58]	93.441±0.610	88.620±2.381	81.034±2.704	73.112±1.922	70.545±0.623	27.684±0.645
A-GEM [10]	92.667±1.352	82.772±5.503	70.579±4.028	60.462±2.001	57.958±0.579	40.969±0.580
ER [75]	94.705±0.790	89.171±2.883	79.962±3.365	71.787±1.608	69.627±0.911	28.749±0.993
DER++ [8]	200	94.904±0.414	91.637±1.871	84.915±2.315	78.373±1.244	76.671±0.391	21.743±0.409
CLS-ER [4]	95.131±0.523	91.421±1.732	84.773±2.665	77.733±1.480	75.609±0.418	22.483±0.456
ESM-ER [80]	95.378±0.531	90.800±2.528	83.438±2.581	76.987±1.219	75.203±0.143	23.564±0.157
UDIL (Ours)	95.097±0.447	93.101±1.305	89.194±1.472	84.704±1.722	82.796±1.882	12.971±2.389
GEM [58]	93.842±0.313	90.663±1.856	85.392±1.856	79.061±1.578	76.619±0.581	21.289±0.579
A-GEM [10]	92.820±1.274	83.564±5.024	72.616±3.865	62.223±2.081	59.654±0.122	39.196±0.171
ER [75]	94.916±0.457	91.491±1.878	86.029±2.176	78.688±1.323	76.794±0.696	20.696±0.744
DER++ [8]	400	95.246±0.228	93.627±1.147	90.011±1.289	85.601±0.982	84.258±0.544	13.692±0.560
CLS-ER [4]	95.233±0.271	92.740±1.268	89.111±1.305	83.678±1.388	81.771±0.354	15.455±0.356
ESM-ER [80]	95.825±0.303	93.378±1.480	89.290±1.604	83.868±1.163	82.192±0.164	16.195±0.150
UDIL (Ours)	95.274±0.469	94.043±0.759	91.511±0.990	87.809±0.849	86.635±0.686	8.506±1.181
GEM [58]	94.212±0.322	92.482±1.125	89.191±1.346	84.866±1.317	82.772±1.079	14.781±1.104
A-GEM [10]	92.902±1.194	84.611±4.451	75.150±3.421	64.510±2.437	61.240±1.026	37.528±1.089
ER [75]	95.144±0.281	92.997±1.195	89.319±1.365	84.352±1.681	81.877±1.157	15.285±1.196
DER++ [8]	800	95.496±0.261	94.960±0.568	93.013±0.689	90.820±0.687	89.746±0.356	7.821±0.371
CLS-ER [4]	95.462±0.174	93.927±0.881	91.275±0.930	87.816±0.988	86.418±0.215	10.598±0.228
ESM-ER [80]	96.086±0.361	94.746±0.915	92.393±0.974	89.745±0.712	88.662±0.263	9.409±0.255
UDIL (Ours)	95.354±0.480	94.711±0.563	92.776±0.695	90.399±0.755	89.191±0.685	6.351±1.304
Joint (Oracle)∞	-	-	-	-	97.150±0.036	-
Table 7: Performances (\%) evaluated on Seq-CORe50. Avg. Acc. and Forgetting are reported to measure the methods' performance. “↑” and “↓” mean higher and lower numbers are better, respectively. We use boldface and underlining to denote the best and the second-best performance, respectively. We use “-” to denote “not appliable” and “★” to denote out-of-memory (OOM) error when running the experiments.
Method	Buffer	D1:3	D4:6	D7:9	D10:11	Overall
Avg. Acc (↑)	Avg. Acc (↑)	Forgetting (↓)
Fine-tune	-	73.707±13.144	34.551±1.254	29.406±2.579	28.689±3.144	31.832±1.034	73.296±1.399
oEWC [81]	-	74.567±13.360	35.915±0.260	30.174±3.195	28.291±2.522	30.813±1.154	74.563±0.937
SI [105]	-	74.661±14.162	34.345±1.001	30.127±2.971	28.839±3.631	32.469±1.315	73.144±1.588
LwF [52]	-	80.383±10.190	28.357±1.143	31.386±0.787	28.711±2.981	31.692±0.768	72.990±1.350
GEM [58]	79.852±6.864	38.961±1.718	39.258±2.614	36.859±0.842	37.701±0.273	22.724±1.554
A-GEM [10]	80.348±9.394	41.472±3.394	43.213±1.542	39.181±3.999	43.181±2.025	33.775±3.003
ER [75]	90.838±2.177	79.343±2.699	68.151±0.226	65.034±1.571	66.605±0.214	32.750±0.455
DER++ [8]	500	92.444±1.764	88.652±1.854	80.391±0.107	78.038±0.591	78.629±0.753	21.910±1.094
CLS-ER [4]	89.834±1.323	78.909±1.724	70.591±0.322	★	★	★
ESM-ER [80]	84.905±6.471	51.905±3.257	53.815±1.770	50.178±2.574	52.751±1.296	25.444±0.580
UDIL (Ours)	98.152±1.665	89.814±2.302	83.052±0.151	81.547±0.269	82.103±0.279	19.589±0.303
GEM [58]	78.717±4.831	43.269±3.419	40.908±2.200	40.408±1.168	41.576±1.599	18.537±1.237
A-GEM [10]	78.917±8.984	41.172±4.293	44.576±1.701	38.960±3.867	42.827±1.659	33.800±1.847
ER [75]	90.048±2.699	84.668±1.988	77.561±1.281	72.268±0.720	72.988±0.566	25.997±0.694
DER++ [8]	1000	89.510±5.726	92.492±0.902	88.883±0.794	86.108±0.284	86.392±0.714	13.128±0.474
CLS-ER [4]	92.004±0.894	85.044±1.276	★	★	★	★
ESM-ER [80]	85.120±4.339	54.852±5.511	61.714±1.840	55.098±3.834	58.932±0.959	20.134±0.643
UDIL (Ours)	98.648±1.174	93.447±1.111	90.545±0.705	87.923±0.232	88.155±0.445	12.882±0.460
Joint (Oracle)	∞	-	-	-	-	99.137±0.049	-