SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1: Initialize:  $\vartheta \gets \vartheta_0, \omega \gets \omega_0, \hat{\mathbb{E}}[S] \gets \mathbf{0}$
2: for  $t \gets 1$  to  $T$  do
3: for  $e\gets 1$  to  $E$  do
4: for  $i\gets 1$  to  $\lfloor |\mathcal{D}_t| / N\rfloor$  do
5:  $B_{i} = (X_{i},Y_{i})\gets \mathrm{Sample}(\mathcal{D}_{t},\{\mathcal{M}_{k}\}_{k = 1}^{t - 1})$
6:  $\mathcal{M}\gets$  UpdateMemory  $(\mathcal{M},B_i)$
7:  $\hat{a}^{(0)}\gets X_i$
8: for  $\ell \gets 1$  to  $L$  do
9:  $a^{(\ell)}\gets f_{\vartheta^{(\ell)}}(\hat{a}^{(\ell -1)})$
0: Compute  $\hat{\mathbb{E}}[S|a^{(\ell)}]$  via Eq. (13) and  $\eta_{i}$  via Eq. (16)
1: Update  $\tilde{\mathbb{E}}[S]$  via Eq. (4)
2: Compute  $a^{\prime (\ell)}$  via Eq. (8) and  $\hat{a}^{(\ell)}$  via Eq. (1)
3:  $\hat{Y}_i = f_{\vartheta (L + 1)}(\hat{a}^{(L)})$
4:  $\vartheta \gets \vartheta -\nabla_{\vartheta}\mathcal{L},\omega \gets \omega -\nabla_{\omega}\mathcal{L}$
5: return  $(\vartheta, \omega)$
For each task
For each epoch
For each batch
For each layer
4.1 Adaptive Balance of BN Statistics for Training
Theorem 5 demonstrates that the benefit of BN for optimization and generalization is potentially interfered in continual learning. To overcome this challenge, here we propose a Bayesian-based strategy for learning an appropriate way of weighting task contributions in normalization statistics. We assume the statistics of batch  $m$  for a certain layer follow a probability distribution  $S \sim \mathcal{P}_{\theta}(S|a_m)$ , where the randomness originates from the neural network parameters  $\theta$ , and then use the conditional expectation w.r.t. the distribution of statistics to calculate the normalized representation:
$$
a _ {m} ^ {\prime} = \frac {a _ {m} - \mathbb {E} [ \mu | a _ {m} ]}{\sqrt {\mathbb {E} [ \sigma^ {2} | a _ {m} ] + \varepsilon}}. \tag {8}
$$
Specifically,  $\mathbb{E}[\mu |a_m]$  and  $\mathbb{E}[\sigma^2 |a_m]$  are the two elements of  $\mathbb{E}[S|a_m]$  as described in Eq. (3):
$$
\mathbb {E} [ S | a _ {m} ] = \sum_ {S} \sum_ {t} \mathcal {P} _ {\theta} (S | t, a _ {m}) \mathcal {P} _ {\theta} (t | a _ {m}) S ^ {t}, \tag {9}
$$
where  $\mathcal{P}_{\theta}(S|t,a_m)$  is the distribution of statistics given the task identity, and  $\mathcal{P}_{\theta}(t|a_m)$  is the likelihood of the task identity occurring in  $a_{m}$ . We treat  $\mathcal{P}_{\theta}(t|a_m)$  as a categorical distribution over the task identity, i.e.,  $\mathcal{P}_{\theta}(t|a_m) = N_t / N$ . Then, BN obtains its normalization statistics by the point estimate of the conditional expectation:
$$
\hat {\mathbb {E}} [ S | a _ {m} ] = \sum_ {t} \mathcal {P} _ {\theta} (t | a _ {m}) S _ {m} ^ {t} = S _ {m}. \tag {10}
$$
As analyzed in Section 3.3, it is critical to consider adaptability of the model parameters  $\theta$  together with normalization statistics in model training, so as to enhance the gradient-representation correlation. Inspired by the theoretical works in Bayesian Inference [22, 12], we propose to capture the variability of  $\theta$  through modeling the task distribution with a Bayesian principle, thus enhancing the benefit of BN for continual learning. Specifically, we assume that there is a random variable  $\tau \in \Delta^{T-1}$  modeling the probability distribution of tasks, where  $\Delta^{T-1}$  denotes the  $T-1$  simplex, i.e.,  $\sum_{t}^{T} \tau_{t} = 1$  and  $\tau_{t} > 0, \forall t$ . Applying the Bayes' rule, we have
$$
\mathbb {E} [ S | a _ {m} ] = \sum_ {S} \sum_ {\tau} \sum_ {t} \mathcal {P} _ {\theta} (S | \tau , t, a _ {m}) \mathcal {P} _ {\theta} (\tau | a _ {m}) \mathcal {P} _ {\theta} (t | \tau , a _ {m}) S ^ {t}, \tag {11}
$$
where  $\mathcal{P}_{\theta}(S|\tau, t, a_m)$  and  $\mathcal{P}_{\theta}(t|\tau, a_m)$  are defined analogous to  $\mathcal{P}_{\theta}(S|t, a_m)$  and  $\mathcal{P}_{\theta}(t|a_m)$  in Eq. (9).  $\mathcal{P}_{\theta}(\tau|a_m)$  serves as the prior captured from the model parameters  $\theta$ , which is generally intractable. For ease of computation, we approximate it with a Dirichlet distribution, i.e., the conjugate prior of the categorical distribution, which is parameterized by the concentration parameter  $\phi \in \mathbb{R}^{T+}$ :
$$
\mathcal {P} _ {\phi} (\tau | a _ {m}) = \frac {\Gamma (\bar {\phi})}{\prod_ {t} \Gamma (\phi_ {t})} \prod_ {t = 1} ^ {T} \tau_ {t} ^ {\phi_ {t} - 1}, \tag {12}
$$
where  $\Gamma (\cdot)$  is the Gamma function, and  $\bar{\phi} = \sum_{t}\phi_{t}$ . By taking advantage of algebraic convenience of the conjugate distribution, we can estimate the conditional expectation by a closed-form expression
$$
\hat {\mathbb {E}} [ S | a _ {m} ] = \sum_ {\tau} \sum_ {t} \mathcal {P} _ {\phi} (\tau | a _ {m}) \mathcal {P} _ {\theta} (t | \tau , a _ {m}) S _ {m} ^ {t} = \sum_ {t} \frac {\phi_ {t} + N _ {t}}{\bar {\phi} + N} S _ {m} ^ {t}. \tag {13}
$$
Intuitively, the concentration parameter  $\phi$  plays the role of adding pseudo-observations to the task weighting procedure, which allows the normalization statistics to adaptively balance task-wise contributions by considering the current state of  $\theta$ . To optimize  $\phi$ , we introduce a learnable parameter  $\psi \in \mathbb{R}^T$  and obtain  $\phi = \exp(\psi)$ . It is worth noting that the dimension of  $\phi$  is incremental with the number of seen tasks, so there is no need to know the total number of tasks in advance. As  $\phi$  is expected to capture the information of  $\theta$ , we propose a regularization term to minimize the Euclidean distance between the adaptively balanced statistics and the estimated population statistics via Eq. (4) as follows:
$$
\mathcal {L} _ {a d a} ^ {(\ell)} (\theta , \psi) = \| \hat {\mathbb {E}} [ S | a _ {m} ] - \hat {\mathbb {E}} [ S _ {m} ] \| ^ {2}, \tag {14}
$$
where  $\hat{\mathbb{E}}[S_m]$  is an improved version introduced in Section 4.2. Eq. (14) provides  $\phi$  with historical information across batches by aligning the population statistics, which contributes to obtaining statistics as good as joint training (detailed in Section 5). From Fig. 1 we observe that adding this regularization term can significantly improve the gradient similarity, leading to stabilized training dynamics. We optimize  $\psi$  and  $\theta$  (with  $\gamma$  and  $\beta$  inside) simultaneously, and the final objective function can be written as
$$
\min  _ {\theta , \psi} \mathcal {L} _ {C L} (\theta) + \lambda \sum_ {\ell} \mathcal {L} _ {a d a} ^ {(\ell)} (\theta , \psi), \tag {15}
$$
where  $\mathcal{L}_{\mathrm{CL}}$  is the loss function of continual learning (e.g., a cross-entropy loss over  $\mathcal{D}_t$  and  $\{\mathcal{M}_k\}_{k=1}^{t-1}$  for classification tasks).  $\lambda$  is a hyperparameter to balance the two terms.
4.2 Adaptive Balance of BN Statistics for Testing
Inspired by Corollaries 2 to 4, we propose to employ an appropriate  $\eta$  function to strike an adaptive balance between EMA and CMA. Concretely, we design a modified momentum  $\eta : [1, m_T] \times (0, 1) \to (0, 1)$  satisfying the following recurrence relation:
$$
\eta_ {i} := \eta (i, \eta_ {i - 1}) = \frac {\eta_ {i - 1}}{\eta_ {i - 1} + (1 - \tilde {\eta}) ^ {\kappa}}, \quad \eta_ {0} := \tilde {\eta} ^ {\kappa}, \tag {16}
$$
where  $\tilde{\eta} \in (0,1)$  is the modified momentum of EMA and  $\kappa \in [0,1]$  is an additional hyperparameter to control the balance of normalization statistics. In particular, Eq. (16) degenerates to EMA when  $\kappa \rightarrow 1$  and to CMA when  $\kappa \rightarrow 0$ . This allows us to find an inflection point for adaptive balance of statistical weights, so as to enjoy the benefits of both EMA and CMA.
In this section, we first briefly introduce the experimental setups of continual learning, and then present the experimental results to validate the effectiveness of our approach.
Table 2: Performance of online task-incremental learning with batch size  $|B| = 10$ . We report the final average accuracy of all seen tasks (↑) with ± standard deviation.
Method	Split CIFAR-10	Split CIFAR-100	Split Mini-ImageNet
|M|=500	|M|=2000	|M|=2000	|M|=5000	|M|=2000	|M|=5000
ER-ACE w/ BN	86.34±2.35	89.17±1.21	61.21±1.63	63.83±1.94	63.62±3.14	64.60±1.73
ER-ACE w/ LN	78.01±6.78	81.59±4.39	42.29±0.29	44.32±0.35	45.91±1.63	45.82±1.91
ER-ACE w/ IN	84.57±1.42	86.03±1.99	49.46±2.25	50.15±1.71	41.07±1.66	40.93±4.38
ER-ACE w/ GN	81.46±3.74	82.85±2.14	44.84±1.41	42.50±2.06	43.45±3.29	45.43±0.62
ER-ACE w/ CN	88.32±1.43	90.01±0.95	61.00±1.58	63.42±0.65	64.23±1.22	65.14±1.48
ER-ACE w/ Ours	88.74±1.77	90.84±2.01	63.88±1.58	67.01±2.90	66.78±1.91	68.03±0.41
DER++ w/ BN	87.61±1.67	90.42±1.83	65.53±1.17	66.23±0.94	61.70±2.40	61.28±1.29
DER++ w/ LN	81.35±4.25	82.80±5.48	43.24±0.86	44.42±2.90	40.05±3.67	37.64±4.26
DER++ w/ IN	84.81±2.99	87.04±2.60	47.39±1.46	49.17±2.13	32.88±1.16	35.23±6.11
DER++ w/ GN	82.05±1.24	83.34±3.34	43.54±1.47	45.12±1.15	40.65±1.76	38.26±3.26
DER++ w/ CN	86.92±0.89	89.75±0.76	66.20±0.38	67.39±1.88	65.09±1.76	66.14±1.40
DER++ w/ Ours	90.15±2.52	91.99±0.81	70.33±0.49	71.70±0.84	66.42±3.62	69.12±2.51
Figure 2: Performance improvements (i.e.,  $\Delta$  FAA) for online continual learning over different batch sizes  $|B|$  and memory sizes  $|\mathcal{M}|$ .
Benchmark: We mainly focus on the setting of online continual learning [30], which is extremely challenging but realistic in application, and further consider the setting of offline continual learning. Here we evaluate three benchmarks that are widely used in both settings. Specifically, Split CIFAR-10 [14] includes 10-class images of size  $32 \times 32$ , randomly split into 5 disjoint tasks with 2 classes each. Split CIFAR-100 [14] includes 100-class images of size  $32 \times 32$ , randomly split into 20 disjoint tasks with 5 classes each. Split Mini-ImageNet [26] includes 100-class images of size  $84 \times 84$ , randomly split into 20 disjoint tasks with 5 classes each. We consider both Task-IL and Class-IL, depending on whether the task identity is provided at test time [25].
Baseline: Following the previous work [19], we employ ER-ACE [7] and  $\mathrm{DER}++$  [6] as the baseline approaches for continual learning, both of which rely on a small memory buffer  $\mathcal{M}$  to replay old training samples. We further consider LiDER [4] in offline scenarios, a recent approach for improving experience replay through regularizing Lipschitz continuity. We then compare our approach to three categories of normalization layers: (1) BN [11], which is the most popular choice that exhibits strong continual learning performance [19, 8]; (2) BN's alternatives designed for single-task learning, such as LN [2], IN [24] and GN [33] (GN can be seen as a general version of LN and IN); and (3) CN [19], which is the state-of-the-art baseline that improves BN for online continual learning.
Implementation: We follow the official implementation as many previous works [19, 4, 7, 6], derived from the mammoth repository [5]. Specifically, we use a ResNet-18 backbone and train all baselines with an SGD optimizer of learning rate 0.03 for Split CIFAR-10/-100 and 0.1 for Split Mini-ImageNet. Notably, we find the official implementation of CN [19] employs different batch sizes (either 10 or 32) for different online continual learning benchmarks. We argue that using a smaller batch size is more "online" and more challenging, as fewer training samples of the current task can be accessed at a time. Therefore, we reproduce all results of online continual learning with one epoch and a small batch size of 10, and further evaluate the effect of using a larger batch size of 32. Besides, we use 50 epochs with a batch size of 32 for offline continual learning. We adopt reservoir sampling to save
Table 3: Performance of online class-incremental learning with batch size  $|B| = 10$ . We report the final average accuracy of all seen classes (↑) with ± standard deviation.
Method	Split CIFAR-10	Split CIFAR-100	Split Mini-ImageNet
|M|=500	|M|=2000	|M|=2000	|M|=5000	|M|=2000	|M|=5000
ER-ACE w/ BN	48.86±2.87	52.25±2.55	24.55±1.37	26.07±2.78	14.41±1.31	14.53±0.58
ER-ACE w/ LN	32.29±0.37	36.64±1.92	11.50±0.10	11.96±1.13	6.28±1.08	6.21±1.07
ER-ACE w/ IN	44.16±1.98	45.86±3.25	15.20±0.87	16.15±0.80	5.00±0.82	5.13±1.48
ER-ACE w/ GN	38.97±2.04	39.60±3.99	12.25±0.83	11.86±0.74	5.33±0.83	6.29±0.60
ER-ACE w/ CN	50.54±4.94	53.97±1.20	23.87±1.14	27.23±0.37	15.76±0.70	15.39±1.45
ER-ACE w/ Ours	52.51±3.04	59.93±4.49	25.39±1.40	28.15±3.24	16.41±0.96	17.08±1.06
DER++ w/ BN	52.85±4.34	57.96±2.57	18.84±1.25	17.26±1.91	7.04±1.47	6.59±0.67
DER++ w/ LN	34.08±2.59	35.37±6.18	6.42±0.95	6.25±0.60	2.94±1.08	2.21±0.43
DER++ w/ IN	45.21±1.88	50.86±2.61	8.52±0.27	8.58±0.46	1.88±0.33	2.33±0.83
DER++ w/ GN	39.54±1.47	38.48±1.13	7.44±0.78	6.77±0.52	2.80±0.18	2.72±0.54
DER++ w/ CN	55.28±1.06	60.25±3.39	18.84±0.09	18.95±1.05	10.16±1.07	8.94±0.94
DER++ w/ Ours	59.44±2.65	64.83±1.89	23.79±2.23	24.12±1.63	10.80±2.76	10.85±1.58
Split CIFAR-100, DER++,  $|\mathcal{M}| = 5000$
Figure 3: Hyperparameter analysis of online task-incremental learning and class-incremental learning for ablation study.
old training samples, which simulates the distribution of incoming batches, and further consider ring buffer that enforces a balanced number for each class [30]. The memory buffer is sized differently to evaluate its impact. The group number of GN and CN is set to 32 as [19]. All results are averaged over three runs with different random seeds and task splits. Please see Appendix C for more details.
Overall Performance: We first evaluate our approach for online continual learning, including the settings of online Task-IL in Table 2 and online Class-IL in Table 3. Consistent with reported results [19, 8], BN performs much better than its alternatives designed for single-task learning (i.e., GN, LN and IN), and is moderately improved by the state-of-the-art CN. In contrast, our approach can improve the performance of BN by a large margin (e.g., up to  $7.68\%$ ,  $6.86\%$  and  $4.26\%$  for online Class-IL on Split CIFAR-10, Split CIFAR-100 and Split Mini-ImageNet, respectively), and outperforms CN on a wide range of memory sizes  $|\mathcal{M}|$ . When increasing the batch size  $|B|$  from 10 to 32, which introduces more training samples at once and makes online continual learning more "offline", we observe that the benefits of CN exhibit variation in many cases, while the benefits of our approach remain consistent and become even stronger (e.g., up to  $15\%$  on Class-IL of Split CIFAR-10, see Fig. 2). Meanwhile, the selection strategies of the memory buffer  $\mathcal{M}$  have no substantial effect, where our approach can provide a similar magnitude of improvement under either reservoir sampling or ring buffer (see Appendix D). We further present the results of offline Task-IL and Class-IL on the hardest Split Mini-ImageNet of the selected benchmarks (see Table 4), where CN even results in negative impacts on BN while our approach still provides strong performance gains in most cases.
Ablation Study & Analysis: We first analyze the introduced two hyperparameters in our approach, i.e.,  $\lambda$  of the regularization term for training and  $\kappa$  of the modified momentum for testing. As shown in Fig. 3, the two hyperparameters can clearly influence the performance of continual learning, which
Table 4: Performance of offline task-incremental learning and class-incremental learning on Split Mini-ImageNet with memory size  $|\mathcal{M}| = 2000$ .
Method	Task-IL	Class-IL
ER-ACE	DER++	LiDER	ER-ACE	DER++	LiDER
BN	36.36±2.33	35.91±1.59	36.96±1.92	10.85±0.54	12.69±1.23	10.99±0.74
CN	35.83±1.43	35.46±1.42	36.23±2.62	10.00±0.62	12.13±0.92	9.97±0.36
Ours	37.64±0.85	36.98±1.52	37.36±0.79	11.05±0.10	13.07±1.30	10.96±0.67
validates the effectiveness of each component in  $\mathrm{AdaB}^2\mathrm{N}$ , and obtain consistent improvements under a wide range of combinations in different settings. Besides, we visualize the statistics parameters of normalization layers in continual learning, in order to explicitly demonstrate how our approach can improve BN (see Fig. 4 for a typical example and Appendix D for all results). We take joint training (JT) of all tasks as the upper bound baseline for continual learning. As more training batches are introduced, it can be obviously seen that the BN statistics are increasingly deviated while the CN statistics are enforced to stabilize. They correspond to the biased strategies of adaptation and balance, respectively, but neither can fit into the curve of JT. In contrast, our approach achieves an adaptive balance of normalization statistics in continual learning, matching closely the upper bound baseline.
In this work, we focus on the most popular BN in continual learning and theoretically analyze its particular challenges for overcoming recency bias. Our analysis demonstrates the dilemma between balance of all tasks and adaptation to the current task, reflected in both training and testing. Following such theoretical insights, we propose Adaptive Balance of BN (AdaB $^2$ N) with a Bayesian-based strategy and a modified momentum to implement BN in a continual learning fashion, which achieves strong performance gains across various scenarios. Interestingly, the robust biological neural networks address a similar dilemma with a combination of normalization mechanisms for neurons, synapses, layers and
networks, some of which are thought to be analogous to the normalization layers in DNNs [23]. More in-depth analysis and modeling of these mechanisms would be a promising direction. We also expect subsequent work to further improve continual learning of other DNN components and overcome potential incompatibilities in optimization, as suggested by our work on normalization layers.
Now we discuss some potential limitations and societal impacts. First, our work applies only to the DNN architectures that employ normalization layers, especially the most popular BN. Second, we need to compute a categorical distribution  $\mathcal{P}_{\theta}(t|a_m)$  over the task identity, which is difficult to adapt directly to the more general setting of task-agnostic continual learning. Third, the prior  $\mathcal{P}_{\theta}(\tau |a_m)$  is approximated as a Dirichlet distribution for ease of computation, which may not be applicable in some practical scenarios. We argue that the broader impact is not obvious at the current stage, as this work is essentially a fundamental research in machine learning.
Figure 4: The norm of BN statistics of a representative layer with  $0.05 \times$  variance in shaded area.
This work was partly supported by the National Natural Science Foundation of China under Grant 62176020, 62106123; the National Key Research and Development Program (2020AAA0106800); the Joint Foundation of the Ministry of Education (8091B042235); the Beijing Natural Science Foundation under Grant L211016; the Fundamental Research Funds for the Central Universities (2019JBZ110); and Chinese Academy of Sciences (OEIP-O-202004). L.W. was also supported by Shimu Tsinghua Scholar.
Table 5 summarizes the notations used in this paper.
Table 5: Notations used in this paper.
Symbol	Meaning
T	Number of tasks
(xt,i,yt,i)	Input samples and its label of task t
Dt	Training set of task t
D1:t-1	Old training sets before task t
M	Memory buffer
Mt	Set of samples from task t saved in memory buffer
B, Bm	m-th batch
N, |B|	Batch size
E	Number of training epochs
mt	Index of the last batch of task t
rm, r(m)	Proportion of task t in Bm
ηm, η(m)	Momentum of exponential moving average (EMA) for Bm
a, am, am(ℓ)	Internal representation of the ℓ-th layer of the m-th batch
a', a'(ℓ), a'(ℓ)	Normalized internal representation am(ℓ)
hat, a(ℓ), a'm	Affined internal representation am(ℓ)
C, C(ℓ)	Number of channels of am(ℓ)
D, D(ℓ)	Feature dimensions of am(ℓ)
μ(·)	Mean function of a normalization layer
σ2(·, ·)	Variance function of a normalization layer
S(·)	Statistic function of a normalization layer to obtain mean and variance
wt	Statistical weight of task t
wtm	Statistical weight of task t within the m-th batch
Sm	Batch statistics of internal representation am
Stm	Batch statistics of task t within internal representation am
ˆE[Sm],ˆE[S|a1,···,am]	Estimated population statistics w.r.t. a1, a2,···, am
E[S|am]	Conditional expectation of batch statistics w.r.t. am
ˆE[S|am]	Estimated conditional expectation of batch statistics w.r.t. am
θ	Parameters of the entire neural network
φ	Parameters of the network layer
ω	Parameters of the normalization layer
γ,β	Parameters of affine transformation
ψ	Parameters of adaptive balance
L	Objective function
Proof of Theorem 1. Let  $g:[1,m_T]\times \mathbb{Z}^{T + }\to [1,T]$  be a function that maps the batch index to its corresponding task index. We abbreviate  $g(i,(m_t)_t^T)$  as  $g(i)$ . Then we have
$$
\begin{array}{l} \hat {\mathbb {E}} \left[ S _ {m _ {T}} \right] = \left(1 - \eta_ {m _ {T}}\right) \hat {\mathbb {E}} \left[ S _ {m _ {T} - 1} \right] + \eta_ {m _ {T}} S _ {m _ {T}} (17) \\ = \left(1 - \eta_ {m _ {T}}\right) \hat {\mathbb {E}} \left[ S _ {m _ {T} - 1} \right] + \eta_ {m _ {T}} r _ {m _ {T}} S _ {m _ {T}} ^ {T} + \frac {\eta_ {m _ {T}} \left(1 - r _ {m _ {T}}\right)}{T - 1} \sum_ {t = 1} ^ {T - 1} S _ {m _ {T}} ^ {t} (18) \\ = \sum_ {i = 1} ^ {m _ {T}} \eta_ {i} r _ {i} \prod_ {j = i + 1} ^ {m _ {T}} (1 - \eta_ {j}) S _ {i} ^ {g (i)} + \sum_ {j = m _ {g (i)} + 1} ^ {m _ {T}} \frac {\eta_ {j} (1 - r _ {j})}{T - 1} \prod_ {k = j + 1} ^ {m _ {T}} (1 - \eta_ {k}) S _ {j} ^ {g (j)} (19) \\ \end{array}
$$
where the first part of Eq. (19) can be considered as the contribution of the "current" tasks of all batches to the running statistics, and the second part can be considered as the contribution of the
"past" tasks. Now we can conclude the proof by noting that the contribution of task  $t$  as the "current" tasks to the running statistics starts from  $m_{t-1} + 1$  to  $m_t$  and the contribution as the "past" task starts from  $m_t + 1$  to  $m_T$ .
Proof of Corollary 2. The first result can be obtained by noting that  $\eta$  is a constant and directly applying the sum formula of geometric series. When  $m_{T} - m_{T - 1} = m_{T - 1} - m_{T - 2} = \dots = m_{1}$ , we have that
$$
w _ {t} = \frac {\bar {\eta} ^ {m _ {T - t}} - \bar {\eta} ^ {m _ {T - t + 1}}}{(1 - \bar {\eta} ^ {m _ {T}}) Z} = \frac {\bar {\eta} ^ {(T - t) m _ {1}} - \bar {\eta} ^ {(T - t + 1) m _ {1}}}{(1 - \bar {\eta} ^ {T m _ {1}}) Z} = \frac {\bar {\eta} ^ {(T - t) m _ {1}} (1 - \bar {\eta} ^ {m _ {1}})}{(1 - \bar {\eta} ^ {T m _ {1}}) Z} \approx \frac {\bar {\eta} ^ {(T - t) m _ {1}}}{Z}
$$
The approximation error is upper bounded by
$$
| \epsilon | = \frac {\bar {\eta} ^ {(T - t) m _ {1}}}{Z} (\frac {1 - \bar {\eta} ^ {m _ {1}}}{1 - \bar {\eta} ^ {m _ {1} T}} - 1) <   \frac {\bar {\eta} ^ {m _ {1}}}{(1 - \bar {\eta} ^ {m _ {1} T}) Z} \leq \mathcal {O} (\frac {\bar {\eta} ^ {m _ {1}}}{1 - \bar {\eta} ^ {m _ {1} T}})
$$
Proof of Corollary 3. Necessity:
Let us start by assuming that  $w_{t} = w_{t + 1}$ . By applying Eq. (6) we have the following
$$
\sum_ {i = m _ {t - 1} + 1} ^ {m _ {t}} \eta_ {i} r _ {i} \prod_ {j = i + 1} ^ {m _ {T}} (1 - \eta_ {j}) = \sum_ {i = m _ {t} + 1} ^ {m _ {t + 1}} \frac {\eta_ {i} \left(r _ {i} T - 1\right)}{T - 1} \prod_ {j = i + 1} ^ {m _ {T}} (1 - \eta_ {j})
$$
Simplifying the equation we obtain that
$$
\sum_ {i = m _ {t - 1} + 1} ^ {m _ {t}} r (1 - \eta) ^ {m _ {T} - i} = \sum_ {i = m _ {t} + 1} ^ {m _ {t + 1}} \frac {r T - 1}{T - 1} (1 - \eta) ^ {m _ {T} - i}
$$
$$
\frac {r}{(1 - \eta) ^ {m _ {t - 1} + 1}} \sum_ {i = 0} ^ {m _ {t} - m _ {t - 1}} \frac {1}{(1 - \eta) ^ {i}} = \frac {r T - 1}{(T - 1) (1 - \eta) ^ {m _ {t} + 1}} \sum_ {i = 0} ^ {m _ {t + 1} - m _ {t}} \frac {1}{(1 - \eta) ^ {i}}
$$
$$
\frac {r (\eta - 1) (1 - \eta) ^ {m _ {t} - m _ {t - 1}} + r}{\eta (1 - \eta) ^ {m _ {t} + 1}} = \frac {(r T - 1) [ (\eta - 1) (1 - \eta) ^ {m _ {t + 1} - m _ {t}} + 1 ]}{\eta (T - 1) (1 - \eta) ^ {m _ {t + 1} + 1}}
$$
$$
r (1 - \bar {\eta} ^ {m _ {t} - m _ {t - 1} + 1}) = \frac {(r T - 1) (1 - \bar {\eta} ^ {m _ {t + 1} - m _ {t} + 1})}{(T - 1) \bar {\eta} ^ {m _ {t + 1} - m _ {t}}}
$$