SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} r - \frac {r (T - T \bar {\eta} ^ {m _ {t + 1} - m _ {t} + 1})}{(T - 1) \bar {\eta} ^ {m _ {t + 1} - m _ {t}} (1 - \bar {\eta} ^ {m _ {t} - m _ {t - 1} + 1})} = \frac {\bar {\eta} ^ {m _ {t + 1} - m _ {t} + 1} - 1}{(T - 1) \bar {\eta} ^ {m _ {t + 1} - m _ {t}} (1 - \bar {\eta} ^ {m _ {t} - m _ {t - 1} + 1})} \\ r = \frac {\bar {\eta} ^ {m _ {t + 1} - m _ {t} + 1} - 1}{\bar {\eta} ^ {m _ {t + 1} - m _ {t}} (1 - T) (\bar {\eta} ^ {m _ {t} - m _ {t - 1} + 1} - 1) + T (\bar {\eta} ^ {m _ {t + 1} - m _ {t} + 1} - 1)} \\ = \frac {\bar {\eta} ^ {m _ {1} + 1} - 1}{\bar {\eta} ^ {m _ {1}} (1 - T) (\bar {\eta} ^ {m _ {1} + 1} - 1) + T (\bar {\eta} ^ {m _ {1} + 1} - 1)} \\ = \frac {1}{T - \bar {\eta} ^ {m _ {1}} (T - 1)} \\ \approx \frac {1}{T} \\ \end{array}
$$
$$
| \epsilon | = \frac {\bar {\eta} ^ {m _ {1}} (T - 1)}{T (T - \bar {\eta} ^ {m _ {1}} (T - 1))} <   \frac {\bar {\eta} ^ {m _ {1}}}{T - \bar {\eta} ^ {m _ {1}} (T - 1)} \leq \mathcal {O} (\bar {\eta} ^ {m _ {1}})
$$
Sufficiency:
$$
w _ {t + 1} - w _ {t} = \sum_ {i = m _ {t} + 1} ^ {m _ {t + 1}} \frac {\eta_ {i} (r _ {i} T - 1)}{T - 1} \prod_ {j = i + 1} ^ {m _ {T}} (1 - \eta_ {j}) - \sum_ {i = m _ {t - 1} + 1} ^ {m _ {t}} \eta_ {i} r _ {i} \prod_ {j = i + 1} ^ {m _ {T}} (1 - \eta_ {j}) \tag {20}
$$
Substituting  $r = 1 / T$  into Eq. (20) yields
$$
\begin{array}{l} w _ {t + 1} - w _ {t} = - \sum_ {i = m _ {t - 1} + 1} ^ {m _ {t}} \frac {\eta_ {i}}{T} \prod_ {j = i + 1} ^ {m _ {T}} (1 - \eta_ {j}) \\ = - \frac {\eta}{T} \sum_ {i = m _ {t - 1} + 1} ^ {m _ {t}} (1 - \eta) ^ {m _ {T} - i} \\ = - \frac {\bar {\eta} ^ {m _ {T} - m _ {t} - 1} \left(1 - \bar {\eta} ^ {m _ {t} - m _ {t - 1} + 1}\right)}{T} \\ \approx 0 \\ \end{array}
$$
$$
| \epsilon | = \frac {\bar {\eta} ^ {m _ {T} - m _ {t} - 1} (1 - \bar {\eta} ^ {m _ {t} - m _ {t - 1} + 1})}{T} <   \frac {\bar {\eta} ^ {m _ {T} - m _ {t} - 1}}{T} \leq \mathcal {O} (\bar {\eta} ^ {m _ {1}})
$$
Proof of Corollary 4. Substituting  $\eta (i) = 1 / (1 + i)$  into Eq. (6)
$$
\begin{array}{l} w _ {t} = \left[ \sum_ {i = m _ {t - 1} + 1} ^ {m _ {t}} \eta_ {i} r _ {i} \prod_ {j = i + 1} ^ {m _ {T}} (1 - \eta_ {j}) + \sum_ {i = m _ {t} + 1} ^ {m _ {T}} \frac {\eta_ {i} (1 - r _ {i})}{T - 1} \prod_ {j = i + 1} ^ {m _ {T}} (1 - \eta_ {j}) \right] / Z \\ = \left[ \sum_ {i = m _ {t - 1} + 1} ^ {m _ {t}} \frac {r _ {i}}{1 + m _ {T}} + \sum_ {i = m _ {t} + 1} ^ {m _ {T}} \frac {1}{T - 1} \frac {1 - r _ {i}}{1 + m _ {T}} \right] / Z \\ = \left[ \frac {r + \frac {1 - r}{T - 1}}{1 + m _ {T}} \right] / Z \\ \end{array}
$$
We can now conclude by noting that  $w_{t}$  is independent of the choice of  $t$ .
The hyperparameter  $\tilde{\eta}$  is fixed to 0.1,  $\kappa$  is selected from  $\{0.1, 0.4, 0.7, 1.0\}$ , and  $\lambda$  is selected from  $\{0.01, 0.1, 1.0, 10.0\}$  for Split CIFAR-10 and Split CIFAR-100 and  $\{0.00001, 0.0001, 0.001, 0.01\}$  for Split Mini-ImageNet. Code is available in the supplementary material and will be released upon acceptance. All experiments are performed on eight NVIDIA RTX A4000 GPUs. The amount of compute is easily affordable, which can be inferred from the running times given in Table 6.
D.1 Time Complexity Analysis
We provide in Table 6 the floating point operations per step (FLOPs/step) of different normalization methods and the total running times (in seconds) under different implementations (the exact calculation of FLOPs can be found in the attached code). It can be seen that compared to BN, our proposed method only slightly increases the computation, while CN almost doubles the computation because it combines both GN (without affine transformation) and BN. Note that the FLOPs only give the theoretical computation; the actual running time depends on the implementation. We measure the actual running time of two normalization implementations: 1) using the plain torch primitive and 2) using the torch nn with cuDNN as the backend. Due to engineering difficulties, we do not currently implement a cuDNN-optimized version of our method. However, it can be seen from Table 6 that on the plain implementation our method only slightly increases the running time, which is consistent with the FLOPs analysis. This suggests that our method has the potential to achieve comparable time complexity as BN through a well-engineered cuDNN implementation.
Table 6: Comparison of FLOPs per step of different normalization layers and total running times (in seconds) under different implementations on Split CIFAR-100 using ER-ACE with  $|B| = 10$  and  $|M| = 2000$  as the baseline approach. - to be exploited.
BN	GN	CN	Ours
FLOPs/step	49.20M	49.18M	86.09M	49.32M
1×	0.99×	1.75×	1.01×
Time (Plain)	430	440	535	489
1×	1.02×	1.24×	1.14×
Time (cuDNN)	324	322	376	-
1×	0.99×	1.16×	-
D.2 Impacts of  $r$
To show how the statistical weights of tasks affect performance, we vary the proportion of the current task in the batch (i.e.,  $r = N_t / N$ ) to different constants (with fixed  $N_t = 10, N = 10, 20, 50, \ldots$ ) and conduct experiments on Split CIFAR-10. It can be seen from Table 7 that the performance first increases rapidly, peaks at around  $r = 1/10$  (i.e., the inverse of the total number of tasks), and then stabilizes. In particular, the accuracy with unfixed  $r$  is comparable with the peak, which supports our Corollary 3. Since the total number of tasks is generally unavailable in continual learning, using an unfixed  $r$  w.r.t  $1/T$  is more practical.
Table 7: Performance comparison of different  $r$  setups on Split CIFAR-10.  $T$  refers to the number of seen tasks that increases over time.
r	1	1/2	1/5	1/10	1/12	1/15	1/20	1/T
Task-IL ACC	23.58	55.65	61.81	62.91	61.49	60.36	58.83	61.61
Class-IL ACC	3.05	17.33	25.26	25.50	25.07	24.04	23.83	24.26
D.3 Impacts of Memory Buffer Selection Strategies
Figure 5: Performance improvements (i.e.,  $\Delta$  FAA) for online continual learning over different selection strategies of memory buffer and different memory sizes  $|\mathcal{M}|$  (with batch size  $|B| = 10$ ).
Fig. 5 shows that our method obtains substantial improvements over both reservoir sampling and ring buffer, which demonstrates the robustness of  $\mathrm{AdaB}^2\mathrm{N}$  to the memory buffer selection strategy.
D.4 Fine-tuning and Joint Training Performance
To establish a simple baseline and upper bound, we present the fine-tuning (FT) and joint training (JT) performance results of different normalization approaches in Table 8. Notably, Ours-FT consistently exhibits better performance than both BN-FT and CN-FT in the absence of replay data. On the other hand, Ours-JT does not outperform BN-JT, indicating that our performance improvements are indeed specific to continual learning.
Table 8: Fine-tuning and joint training performance on Split CIFAR-10 with  $|B| = 10$ .
Fine-tuning	Joint tuning
Task-IL ACC	Class-IL ACC	Task-IL ACC	Class-IL ACC
BN	34.97±2.42	5.82±0.37	76.80±1.48	41.36±2.16
CN	34.48±0.88	4.68±0.59	76.41±0.61	41.46±0.61
Ours	36.31±1.72	5.95±0.88	75.79±1.31	40.14±1.74
D.5 Scalability Analysis
In order to evaluate the scalability of our proposed method, we perform an additional experiment with ImageNet-sub [10], which includes 100 classes randomly selected from ImageNet of size  $224 \times 224$ . We split them into 10 incremental phases for online continual learning (ER,  $|B| = 32$ ,  $|M| = 2000$ ), and reuse the hyperparameters ( $\lambda$  and  $\kappa$ ) of Split Mini-ImageNet. As shown in Table 9, the Task-IL/Class-IL accuracy of our approach is clearly better than BN, demonstrating scalability on larger datasets. Furthermore, we perform experiments focused on online domain-incremental learning (Domain-IL) to evaluate its performance in handling more subtle distribution shifts. Specifically, we employ two datasets: Permuted MNIST [13], which applies random permutations to MNIST digit pixels to create 20 subsequent tasks, and DomainNet [18], comprising 345-class images from six distinct domains. Table 9 shows that our approach can clearly improve batch normalization on both Permuted MNIST and DomainNet, thanks to a better resolution of balance and adaptation.
Table 9: Performance on scaling to larger datasets and handling more subtle distribution shifts.
ImageNet-sub	Permuted MNIST	DomainNet
Task-IL	Class-IL	Domain-IL
BN	72.92	16.86	64.89	7.38
CN	76.48	21.88	65.97	10.80
Ours	77.24	22.08	77.15	7.81
D.6 Statistical Significance
We perform non-parametric paired t-test (i.e., the Wilcoxon signed-rank test) to evaluate whether  $\mathrm{AdaB}^2\mathrm{N}$  exhibits statistically significant superiority over comparable baselines in both online and offline scenarios, as well as a combination of both. The null hypothesis  $H_0$  states that the mean performance difference between  $\mathrm{AdaB}^2\mathrm{N}$  and the alternate baseline is zero for a given scenario, while the alternative hypothesis  $H_{1}$  states a non-zero difference. It is worth noting that we conduct two-tailed tests to avoid making an assumption that  $\mathrm{AdaB}^2\mathrm{N}$  consistently outperforms the baseline. Our samples are collected from all the results presented in the manuscript. The resulting  $p$ -values for these tests, as detailed in Table 10, demonstrate that  $\mathrm{AdaB}^2\mathrm{N}$  outperforms BN and CN statistically significantly across online, offline, and combined online and offline scenarios.
Table 10: Non-parametric paired t-test (the Wilcoxon signed-rank test) on whether  $\mathrm{AdaB}^2\mathrm{N}$  is statistically significantly better than the baselines  $(p < 0.05)$ .
Online	Offline	Both
BN	CN	BN	CN	BN	CN
p-value	8.0e-13	5.2e-11	1.6e-02	1.5e-05	1.0e-14	1.0e-13
Is AdaB2N statistically significantly better?	✓	✓	✓	✓	✓	✓
D.7 Forgetting Measure
Table 11 shows that our method is generally the lowest in terms of forgetting measure.
Table 11: Forgetting measure  $(\downarrow)$  of online task-incremental learning with batch size  $|B| = 10$ . We use bold, underline, and italic to indicate the first, second, and third best results respectively.
Method	Split CIFAR-10	Split CIFAR-100	Split Mini-ImageNet
|M|=500	|M|=2000	|M|=2000	|M|=5000	|M|=2000	|M|=5000
ER-ACE w/ BN	3.31±0.95	1.76±1.84	2.15±1.11	1.71±0.98	3.14±1.42	2.80±1.07
ER-ACE w/ LN	4.03±3.25	0.99±1.43	2.92±0.81	1.46±0.44	2.93±0.08	2.53±0.93
ER-ACE w/ IN	1.74±1.04	1.76±0.96	2.11±0.61	1.41±0.63	3.86±0.59	5.07±1.20
ER-ACE w/ GN	1.30±0.36	1.73±1.61	0.86±0.26	3.21±0.96	4.13±1.98	3.07±1.44
ER-ACE w/ CN	1.67±0.25	0.90±0.51	2.53±0.53	2.03±0.33	3.72±1.26	4.21±1.37
ER-ACE w/ Ours	1.22±0.60	0.84±0.78	1.81±0.66	1.58±1.35	2.21±2.38	1.94±0.87
DER++ w/ BN	2.35±0.73	0.22±0.03	1.26±0.42	1.31±1.07	1.97±0.94	2.60±0.32
DER++ w/ LN	1.61±1.05	0.65±0.22	2.02±1.28	1.88±1.41	3.16±0.70	2.95±0.61
DER++ w/ IN	1.90±0.47	0.68±0.62	2.91±1.89	1.78±1.03	3.46±0.56	3.64±0.40
DER++ w/ GN	1.42±1.03	0.89±1.05	2.32±1.45	1.41±0.63	3.88±1.52	3.70±1.29
DER++ w/ CN	3.89±0.52	2.05±0.40	1.63±0.77	1.07±0.28	4.04±1.17	3.85±1.08
DER++ w/ Ours	1.81±1.74	0.38±0.17	0.92±0.58	0.73±0.41	3.65±1.49	1.36±0.32
D.8 Full Results for Dynamics of Normalization Statistics
We provide the norm dynamics of the batch statistics and population statistics for all normalization layer in Figs. 6 and 7. For better illustration, each curve in the figures indicates the mean after Gaussian smoothing with a kernel size of 20 and the shaded area indicates the  $0.05 \times$  variance. It can be seen that our method roughly tracks the joint training (JT) on many layers (e.g., 2, 3, 5, 7, 8, 12, 15, 17, 18, 19).
Figure 6: Norm dynamics of the batch statistics of all normalization layer.
Figure 7: Norm dynamics of the population statistics of all normalization layer.