SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where the second inequality follows because the ESS is monotonically increasing for  $\alpha$  and  $\alpha \geq nd^2$ . Since  $(1 + \sqrt{nd^2})^{2(k - j)} \geq 1 + 2\sqrt{nd^2} (k - j)$  and  $k - j \geq 1$ , we have that
$$
n _ {j} ^ {\rightleftarrows^ {k}} \geq n _ {j} ^ {\rightarrow} + \frac {n}{\alpha} \frac {\sqrt {n d ^ {2}}}{1 + \sqrt {n d ^ {2}}} \geq n _ {j} ^ {\rightarrow} + n \frac {1}{\phi} \frac {1}{1 + \sqrt {n d ^ {2}}}
$$
because  $\alpha \leq \sqrt{nd^2}\phi$
3. If  $nd^2 \geq 1 \Rightarrow 1 \leq nd^2 \leq \alpha \leq nd^2\phi$  because  $\alpha = nd^2\frac{\sqrt{1 + \frac{4}{nd^2}} + 1}{2}$ , then we have that  $n_j \stackrel{=k}{=} 1$ .
$$
n _ {j} ^ {\rightleftharpoons k} \geq n _ {j} ^ {\rightarrow} + n \frac {1}{\alpha} \frac {2 ^ {2 (k - j)} - 1}{2 ^ {2 (k - j)} + 1} \geq n _ {j} ^ {\rightarrow} + n \frac {1}{n d ^ {2}} \frac {1}{\phi} \frac {3}{5}
$$
where the first inequality follows because the ESS is monotonically increasing for  $\alpha$  and  $\alpha \geq 1$  and the second inequality is obtained using  $k - j \geq 1$  and  $\alpha \leq nd^2\phi$ .
H Additional numerical results and implementation details
In this section we describe the datasets used for the numerical results in Section 5, we provide further details for the numerical experimentations carried out, and include several additional results. Specifically, in the first set of additional results, we evaluate the classification performance of the proposed method in comparison with state-of-the-art techniques for different sample sizes; in the second set of additional results, we further show the performance improvement leveraging information from all the tasks in the sequence with additional datasets; in the third set of additional results, we show the classification error and the running time of IMRCs for different hyper-parameter values; and in the fourth set of additional results, we evaluate the assumption of change between tasks being independent and zero-mean. In addition, in the folder Implementation_IMRC in the supplementary materials we provide the code of the proposed IMRCs with the setting used in the numerical results.
In Section 5, we use 12 publicly available datasets [1, 37, 36, 39, 10, 40, 41, 2], and http://yann.lecun.com/exdb/mnist/. The summary of these datasets is provided in Table 3 that shows the number of classes, the number of samples, and the number of tasks. In the following, we further describe the tasks and the time-dependency of each dataset used.
- The "Yearbook" dataset contains portraits' photographs over time and the goal is to predict males and females. Each task corresponds to portraits from one year from 1905 to 2013.
- The "ImageNet noise" dataset contains images with increasing noise over tasks and the goal is to predict if an image is a bird or a snake. The sequence of tasks corresponds to the noise factors [0.0, 0.4, 0.8, 1.2, 1.6, 2.0, 2.4, 2.8, 3.2, 3.6] [35].
- The "DomainNet" dataset contains six different domains with decreasing realism and the goal is to predict if an image is an airplane, bus, ambulance, or police car. The sequence of tasks corresponds to the six domains: real, painting, infographic, clipart, sketch, and quickdraw.
- The "UTKFaces" dataset contains face images in the wild with increasing age and the goal is to predict males and females. The sequence of tasks corresponds to face images with different ages from 0 to 116 years.
- The "Rotated MNIST" dataset contains rotated images with increasing angles over tasks and the goal is to predict if the number in an image is greater than 5 or not. Each  $j$ -th task corresponds to a rotation angle randomly selected from  $\left[\frac{180(j - 1)}{k}, \frac{180j}{k}\right]$  degrees where  $j \in \{1,2,\dots,k\}$  and  $k$  is the number of tasks.
- The "CLEAR" dataset contains images with a natural temporal evolution of visual concepts and the goal is to predict if an image is soccer, hockey, or racing. Each task corresponds to one year from 2004 to 2014.
- The "Power Supply" dataset contains three year power supply records from 1995 to 1998 and the goal is to predict which hour the current power supply belongs to. We relabel into binary classification according to pm. or am. as in [10].
- The "Usenet" dataset is split into Usenet1 and Usenet2 which both contains a stream of messages from different 20 newsgroups that are sequentially presented to a user and the goal is to predict the personal interests.
- The "German" dataset contains information about people who take a credit by a bank and the goal is to classify each person as good or bad credit risks.
- The "Spam" dataset contains emails and the task is to predict if an email is malicious spam email or legitimate email.
- The "Covertype" dataset contains cartographic variables of a forest area obtained from US Forest Service (USFS) and the goal is to predict the forest cover type.
The samples in each task are randomly split in 100 samples for test and the rest of the samples for training. The samples used for training in the numerical results are randomly sampled from each group of training samples in each repetition.
In Section 5, we compare the results of IMRC methods with 7 state-of-the-art-techniques [5, 17, 4, 6, 9, 10, 16]. In the following, we briefly describe each method used.
- GEM method [5] is a technique developed for continual learning. The method provided by Lopez-Paz & Ranzato learns each new task using a stochastic gradient descent with inequality constraints given by the losses of preceding tasks. Such constraints avoid the increase of the loss of each preceding tasks.
- MER method [17] is a technique developed for continual learning. The method provided by Riemer et al. learns each new task using sample sample sets that include random samples of preceding tasks. Such samples of preceding tasks are stored in a memory buffer.
- ELLA method [4] is a techniques developed for continual learning. The method provided by Ruvolo & Eaton learns each new task transferring knowledge from a shared basis of task models.
- EWC method [6] is a technique developed for continual learning. The method provided by Kirkpatrick et al. learns each new task regularizing the loss with regularization parameters given by the Fisher information.
- Condor method [9] is a technique developed for concept drift adaptation. The method provided by Zhao et al. is an ensemble method that adapts to evolving tasks by learning weighting the models in the ensemble at each time step.
Table 3: Datasets characteristics.
Dataset	Classes	Samples	Tasks
Yearbook [1]	2	37,921	126
ImageNet Noise [35]	2	12,000	10
DomainNet [36]	4	6,256	6
UTKFace [37]	2	23,500	94
Rotated MNIST [38]	2	70,000	60
CLEAR [39]	3	10,490	10
Power Supply [10]	2	29,928	99
Usenet1 [40]	2	1,500	5
Usenet2 [40]	2	1,500	5
German [41]	2	1,000	3
Spam [2]	2	6,213	20
Covertype [10]	2	581,012	1,936
(a) Classification error per number of tasks using "DomainNet" dataset.
(b) Classification error per sample size using "DomainNet" dataset.
Figure 5: Forward and backward learning can sharply boost performance and ESS as tasks arrive.
- DriftSurf method [10] is a technique developed for concept drift adaptation. The method provided by Tahmasbi et al. adapts to evolving tasks by using a drift detection method. Such method allows to restart a new model when a change in the distribution is detected.
- AUE method [16] is a technique developed for concept drift adaptation. The method provided by Brzezinski & Stefanowski is an ensemble method that adapts to evolving tasks by incrementally updating all classifiers in the ensemble and weighting them with non-linear error functions.
The classifier parameters in the numerical results are obtained using an accelerated subgradient method based on Nesterov approach [32, 33]. Such subgradient method applied to optimization (3) obtains at each step classifier parameters  $\mu$  from the mean and confidence vectors  $\tau, \lambda$  using the iterations for  $l = 1, 2, \ldots, K$
$$
\bar {\boldsymbol {\mu}} (l + 1) = \boldsymbol {\mu} (l) + a _ {l} \left(\boldsymbol {\tau} - \partial \varphi (\boldsymbol {\mu} (l)) - \lambda \operatorname {s i g n} (\boldsymbol {\mu} (l))\right) \tag {22}
$$
$$
\boldsymbol {\mu} (l + 1) = \bar {\boldsymbol {\mu}} (l + 1) + \theta_ {l + 1} \left(\theta_ {l} ^ {- 1} - 1\right) (\boldsymbol {\mu} (l) - \bar {\boldsymbol {\mu}} (l))
$$
where  $\mathrm{sign}(\cdot)$  denotes the sign function,  $\pmb{\mu}(l)$  is the  $l$ -th iterate for  $\pmb{\mu}$ ,  $\theta_l = 2/(l+1)$  and  $a_l = 1/(l+1)^{3/2}$  are the step sizes and  $\partial \varphi(\pmb{\mu}(l))$  denotes a subgradient of  $\varphi(\cdot)$  at  $\pmb{\mu}(l)$  with
$$
\varphi (\boldsymbol {\mu}) = \max  _ {x \in \mathcal {X}, \mathcal {C} \subseteq \mathcal {Y}} \frac {\sum_ {y \in \mathcal {C}} \Phi (x , y) ^ {\mathrm {T}} \boldsymbol {\mu} - 1}{| \mathcal {C} |}.
$$
In addition, the above subgradient method is implemented using  $K = 2000$  iterations and a warm-start that initializes the classifier parameters in (22) with the solution obtained for the closest task.
Table 4: Classification error and standard deviation of the proposed IMRC method in comparison with the existing techniques for  $n = 50$  samples per task.
Method	GEM	MER	ELLA	EWC	Condor	DriftSurf	AUE	IMRC
Yearbook	.16 ± .02	.11 ± .01	.43 ± .10	.38 ± .02	.11 ± .02	.29 ± .02	.29 ± .02	.10 ± .02
I. noise	.16 ± .06	.10 ± .01	.47 ± .05	.47 ± .05	.10 ± .02	.48 ± .02	.48 ± .02	.10 ± .02
DomainNet	.65 ± .05	.29 ± .02	.67 ± .05	.75 ± .05	.45 ± .01	.32 ± .00	.32 ± .00	.28 ± .03
UTKFaces	.12 ± .00	.11 ± .09	.18 ± .11	.12 ± .00	.14 ± .02	.12 ± .00	.12 ± .00	.10 ± .00
R. MNIST	.29 ± .04	.41 ± .13	.48 ± .05	.44 ± .01	.43 ± .01	.48 ± .02	.48 ± .02	.25 ± .02
CLEAR	.08 ± .01	.09 ± .04	.60 ± .05	.64 ± .03	.35 ± .01	.33 ± .00	.33 ± .00	.05 ± .02
P. Supply	.47 ± .03	.47 ± .02	.35 ± .01	.47 ± .02	.36 ± .01	.40 ± .01	.40 ± .01	.28 ± .01
Usenet1	.47 ± .02	.46 ± .02	.32 ± .01	.48 ± .02	.45 ± .02	.39 ± .01	.39 ± .01	.30 ± .01
Usenet2	.35 ± .01	.32 ± .02	.32 ± .01	.42 ± .05	.35 ± .02	.27 ± .01	.30 ± .01	.32 ± .01
German	.34 ± .01	.33 ± .01	.31 ± .01	.30 ± .01	.34 ± .01	.30 ± .01	.30 ± .01	.28 ± .01
Spam	.07 ± .01	.09 ± .02	.23 ± .03	.23 ± .03	.19 ± .01	.32 ± .01	.33 ± .01	.08 ± .00
Covert.	.09 ± .00	.08 ± .00	.11 ± .01	.08 ± .00	.07 ± .02	.09 ± .00	.10 ± .00	.08 ± .00
Table 5: Classification error and standard deviation of the proposed IMRC method in comparison with the existing techniques for  $n = 100$  samples per task.
Method	GEM	MER	ELLA	EWC	Condor	DriftSurf	AUE	IMRC
Yearbook	.17 ± .03	.10 ± .01	.43 ± .02	.27 ± .06	.09 ± .01	.27 ± .02	.27 ± .01	.08 ± .02
I. noise	.13 ± .07	.10 ± .01	.47 ± .04	.46 ± .06	.09 ± .01	.48 ± .02	.48 ± .02	.09 ± .01
DomainNet	.53 ± .10	.26 ± .04	.67 ± .05	.74 ± .05	.44 ± .00	.32 ± .00	.32 ± .00	.28 ± .01
UTKFaces	.12 ± .00	.11 ± .01	.17 ± .11	.12 ± .00	.11 ± .01	.12 ± .00	.12 ± .00	.10 ± .00
R. MNIST	.28 ± .02	.45 ± .10	.47 ± .05	.40 ± .01	.41 ± .01	.48 ± .02	.48 ± .02	.21 ± .00
CLEAR	.09 ± .02	.05 ± .02	.60 ± .05	.62 ± .04	.35 ± .01	.33 ± .00	.34 ± .00	.05 ± .02
P. Supply	.47 ± .02	.48 ± .03	.37 ± .04	.46 ± .00	.36 ± .00	.37 ± .02	.38 ± .02	.25 ± .00
Usenet1	.46 ± .02	.46 ± .02	.35 ± .05	.47 ± .02	.40 ± .01	.39 ± .01	.39 ± .01	.30 ± .01
Usenet2	.34 ± .00	.29 ± .01	.35 ± .02	.35 ± .02	.32 ± .02	.26 ± .02	.27 ± .02	.30 ± .00
German	.34 ± .01	.29 ± .01	.29 ± .01	.30 ± .01	.30 ± .01	.29 ± .00	.32 ± .01	.29 ± .00
Spam	.07 ± .01	.08 ± .01	.26 ± .05	.17 ± .02	.10 ± .02	.32 ± .01	.33 ± .01	.08 ± .00
Covert.	.08 ± .00	.08 ± .00	.11 ± .00	.08 ± .00	.07 ± .02	.09 ± .00	.09 ± .00	.08 ± .00
Table 6: Classification error and standard deviation of the proposed IMRC method in comparison with the existing techniques for  $n = 150$  samples per task.
Method	GEM	MER	ELLA	EWC	Condor	DriftSurf	AUE	IMRC
Yearbook	.16 ± .03	.10 ± .01	.43 ± .08	.22 ± .02	.09 ± .01	.23 ± .01	.23 ± .01	.08 ± .01
I. noise	.12 ± .03	.07 ± .01	.47 ± .04	.45 ± .07	.09 ± .01	.48 ± .02	.48 ± .02	.08 ± .01
DomainNet	.49 ± .10	.28 ± .02	.67 ± .05	.74 ± .05	.43 ± .01	.32 ± .00	.32 ± .00	.27 ± .02
UTKFaces	.12 ± .00	.11 ± .01	.17 ± .11	.12 ± .00	.10 ± .01	.12 ± .00	.12 ± .00	.10 ± .00
R. MNIST	.27 ± .01	.47 ± .05	.47 ± .05	.38 ± .01	.41 ± .02	.48 ± .02	.48 ± .02	.20 ± .01
CLEAR	.08 ± .01	.05 ± .02	.60 ± .04	.60 ± .04	.36 ± .01	.33 ± .00	.33 ± .00	.04 ± .01
P. Supply	.47 ± .01	.48 ± .02	.30 ± .01	.47 ± .01	.36 ± .01	.36 ± .00	.36 ± .01	.24 ± .00
Usenet1	.47 ± .02	.41 ± .03	.29 ± .01	.48 ± .01	.43 ± .02	.39 ± .00	.39 ± .01	.29 ± .01
Usenet2	.34 ± .00	.29 ± .00	.26 ± .01	.32 ± .01	.32 ± .02	.26 ± .00	.28 ± .00	.31 ± .00
German	.34 ± .01	.30 ± .00	.25 ± .01	.30 ± .02	.31 ± .01	.29 ± .01	.29 ± .01	.28 ± .01
Spam	.07 ± .01	.07 ± .01	.23 ± .03	.13 ± .02	.11 ± .02	.32 ± .01	.33 ± .01	.07 ± .00
Covert.	.08 ± .00	.08 ± .00	.10 ± .01	.08 ± .00	.07 ± .02	.09 ± .00	.09 ± .00	.08 ± .00
(a) Single task.
(b) Forward.
(c) Forward and backward.
Figure 6: Forward and backward learning can improve performance of preceding tasks.
Table 7: Classification error of the proposed IMRC method varying  $W$  and  $b$  .
Hyper-parameter	W=2	W=4	W=6	b=1	b=2	b=3	b=4	b=5
Sample size n	10	100	10	100	10	100	10	100	10	100	10	100	10	100	10	100
Yearbook	.13	.08	.13	.09	.13	.09	.14	.10	.09	.14	.13	.08	.13	.08	.13	.08
ImageNet noise	.15	.09	.15	.09	.15	.09	.15	.09	.15	.09	.15	.09	.15	.08	.15	.08
DomainNet	.34	.28	.32	.27	.33	.28	.36	.29	.35	.28	.34	.28	.34	.28	.34	.28
UTKFaces	.10	.10	.10	.10	.10	.10	.10	.10	.10	.10	.10	.10	.10	.10	.10	.10
Rotated MNIST	.36	.21	.35	.21	.36	.21	.36	.22	.36	.22	.36	.21	.36	.21	.36	.21
CLEAR	.09	.05	.09	.05	.09	.06	.10	.05	.09	.05	.09	.05	.09	.05	.08	.05
Weather	.31	.31	.31	.31	.31	.31	.31	.31	.31	.31	.31	.31	.31	.31	.31	.31
Power Supply	.30	.25	.30	.25	.30	.25	.32	.27	.31	.26	.30	.25	.30	.25	.30	.25
Usenet1	.32	.30	.33	.31	.33	.31	.32	.30	.32	.30	.32	.30	.32	.30	.32	.30
Usenet2	.33	.30	.34	.31	.34	.31	.34	.30	.33	.30	.33	.30	.33	.30	.32	.30
German	.34	.29	.34	.29	.34	.29	.34	.29	.34	.29	.34	.29	.34	.29	.34	.29
Spam	.13	.08	.14	.08	.13	.08	.14	.09	.13	.08	.13	.08	.12	.07	.12	.07
Covertype	.08	.08	.08	.08	.08	.08	.08	.08	.08	.08	.08	.08	.08	.08	.08	.08
In the first set of additional results, we further compare the classification error of IMRCs with the state-of-the-art techniques. The results in Table 1 in the paper as well as Tables 4, 5, and 6 are obtained computing the classification error 50 times for each sample size. Table 1 in the paper shows classification errors for  $n = 10$  samples, while Tables 4, 5, and 6 show the classification error for  $n = 50$ ,  $n = 100$ , and  $n = 150$  samples, respectively. As can be observed in Table 4, the performance improvement of IMRCs in comparison with the state-of-the-art techniques for  $n = 50$ ,  $n = 100$ , and  $n = 150$  is similar to that shown in the paper for  $n = 10$ .
In the second set of additional results, we further illustrate the relationship among classification error, number of tasks, and sample size. Figure 4 in the paper as well as Figure 5 are obtained computing the classification error over all the sequences of consecutive tasks of length  $k$  in the dataset. Then, we repeat such experiment 10 times with randomly chosen training sets of size  $n$ . Figure 5 extends the results for IMRCs using "DomainNet" dataset completing those in the main paper that show the results using "Yearbook" dataset. Figure 5a shows the classification error of IMRC method divided by the classification error of single-task learning for different number of tasks with  $n = 10$  and  $n = 100$  sample sizes. In addition, Figure 5b shows the classification error of IMRC method for different sample sizes with  $k = 10$  tasks. Figures 5a and 5b show similar behavior to Figures 4a and 4b in the paper, respectively. In addition, Figure 6 shows the classification error of IMRCs per step and task with single-task learning, forward learning, and forward and backward learning using the "Yearbook" dataset. Such figure shows that forward and backward learning can improve performance of preceding tasks, while forward learning and single task learning maintain the same performance over time.
In the third set of additional results, we further assess the change in classification error and the running time of IMRCs varying the hyper-parameters. Table 7 shows the classification error of IMRCs varying the values of hyper-parameter for the window size  $W$  and the number of backward steps  $b$ , completing those in the paper that show the results for  $W = 2$  and  $b = 3$ . As shown in the table, the proposed IMRCs do not require a careful fine-tuning of hyper-parameters and similar performances are obtained by using different values. In addition, Table 8 shows the mean running
Table 8: Running time of IMRC method in comparison with the state-of-the-art-techniques.
Dataset	GEM	MER	ELLA	EWC	b=1	b=2	b=3	b=4	b=5
Sample size n	10	100	10	100	10	100	10	100	10	100	10	100	10	100	10	100	10	100
Yearbook	0.098	0.476	0.166	3.726	0.066	0.070	0.358	3.031	0.094	0.324	0.105	0.397	0.133	0.487	0.167	0.582	0.184	0.664
ImageNet noise	0.010	0.037	0.079	1.052	0.073	0.073	0.032	0.252	0.259	0.490	0.261	0.531	0.284	0.561	0.304	0.585	0.438	0.601
DomainNet	0.005	0.020	0.066	0.900	0.054	0.065	0.018	0.155	0.518	8.463	0.543	8.983	0.542	9.514	0.559	9.699	0.571	9.921
UTKFaces	0.310	0.180	0.127	3.458	0.059	0.062	0.245	2.246	0.108	0.348	0.115	0.401	0.133	0.488	0.156	0.573	0.190	0.664
Rotated MNIST	0.180	1.094	0.211	4.092	0.176	0.198	0.587	5.296	0.135	0.471	0.165	0.599	0.209	0.737	0.249	0.877	0.288	1.010
CLEAR	0.009	0.034	0.074	1.063	0.077	0.073	0.031	0.248	0.235	1.310	0.252	1.406	0.255	1.469	0.271	1.595	0.360	1.693
Power Supply	0.031	0.169	0.025	0.249	0.007	0.006	0.190	1.937	0.241	0.768	0.257	0.840	0.291	1.025	0.351	1.209	0.425	1.404
Usenet1	0.003	0.014	0.012	0.115	0.008	0.009	0.012	0.117	0.320	0.930	0.324	0.934	0.366	0.946	0.368	0.980	0.400	1.472
Usenet2	0.003	0.013	0.012	0.113	0.008	0.009	0.012	0.117	0.272	1.164	0.288	1.194	0.302	1.224	0.306	1.218	0.322	1.772
German	0.002	0.009	0.011	0.114	0.010	0.011	0.010	0.074	0.260	0.770	0.266	0.773	0.303	0.773	0.466	1.110	0.606	1.796
Spam	0.014	0.061	0.020	0.194	0.060	0.062	0.055	0.552	0.235	1.061	0.265	1.090	0.267	1.242	0.295	1.278	0.321	2.020
Covertype	1.094	3.124	2.194	6.431	0.057	0.060	2.201	6.438	0.139	0.586	0.166	0.697	0.211	0.850	0.244	1.003	0.278	1.15
(a) Yearbook dataset
(b) Yearbook dataset for  $n = 10$
Figure 7: Running time per task varying the sample size, the number of tasks, and the number of backward steps.
(c) Yearbook dataset for  $n = 100$
time per task in seconds of IMRCs for  $b = 1,2,\dots ,5$  backward steps in comparison with the state-of-the-art techniques that learn a sequence of tasks. Such table shows that the methods proposed for backward learning do not require a significant increase in complexity, and the running time of the proposed method is similar to that of other state-of-the-art methods. Figure 7 further assesses the running time of IMRCs for  $b = 1,2,\ldots ,5$  backward steps varying the sample size and the number of tasks. Such figure shows that the running time of IMRCs increases moderately and linearly with the number of backward steps and the number of tasks.
In the fourth set of additional results, we evaluate the assumption of change between tasks being independent and zero-mean by assessing the partial autocorrelation of mean vectors. In particular, the partial autocorrelation at any lag would be zero if tasks are i.i.d.; while the partial autocorrelation at lag 1 is larger than zero if tasks satisfy the assumption of Section 2. Figure 8 shows the averaged partial autocorrelation of the mean vectors components  $+/ -$  their standard deviations for different lags using "Yearbook" and "UTKFaces" datasets. Such figure shows a partial autocorrelation clearly non-zero at lag 1 that reflects dependence between consecutive mean vectors, as described by the assumption of Section 2.
(a) "Yearbook" dataset
(b) "UTKFaces" dataset
Figure 8: Averaged partial autocorrelation of mean vectors components  $+ / -$  their standard deviations.