SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
= \int_ {0} ^ {t} \mathbb {E} _ {q _ {\phi} (z ^ {t} | z ^ {G}, \mathcal {T} ^ {t})} \Big [ \log \frac {p _ {\theta} (Y _ {\mathcal {T}} ^ {t} | X _ {\mathcal {T}} ^ {t} , z ^ {t}) p _ {\theta} (z ^ {t} | \mathcal {C} ^ {t} , z ^ {G}) q _ {\phi} (z ^ {t} | z ^ {G} , \mathcal {T} ^ {t})}{p _ {\theta} (z ^ {t} | \mathcal {T} ^ {t} , z ^ {G}) q _ {\phi} (z ^ {t} | z ^ {G} , \mathcal {T} ^ {t})} \Big ]
$$
(Equivalent fraction)
(11e)
$$
= \int_ {0} ^ {t} \mathbb {E} _ {q _ {\phi} (z ^ {t} | z G, \mathcal {T} ^ {t})} \left[ \log p _ {\theta} \left(Y _ {\mathcal {T}} ^ {t} \mid X _ {\mathcal {T}} ^ {t}, z ^ {t}\right) \right]
$$
$$
- D _ {\mathrm {K L}} \left(q _ {\phi} \left(z ^ {t} \mid z ^ {G}, \mathcal {T} ^ {t}\right) \| p _ {\theta} \left(z ^ {t} \mid z ^ {G}, \mathcal {C} ^ {t}\right)\right) + D _ {\mathrm {K L}} \left(q _ {\phi} \left(z ^ {t} \mid z ^ {G}, \mathcal {T} ^ {t}\right) \| p _ {\theta} \left(z ^ {t} \mid z ^ {G}, \mathcal {T} ^ {t}\right)\right)
$$
(By definition of KL divergence)
(11f)
$$
\geq \int_ {0} ^ {t} \mathbb {E} _ {q _ {\phi} (z ^ {t} | z ^ {G}, \mathcal {T} ^ {t})} \Big [ \log p _ {\theta} (Y _ {\mathcal {T}} ^ {t} | X _ {\mathcal {T}} ^ {t}, z ^ {t}) \Big ] - D _ {\mathrm {K L}} \big (q _ {\phi} (z ^ {t} | z ^ {G}, \mathcal {T} ^ {t}) \| p _ {\theta} (z ^ {t} | z ^ {G}, \mathcal {C} ^ {t}) \big)
$$
$(\because \mathrm{KL}$  divergence  $\geq 0)$
(11g)
Plugging Eq. (11g) into Eq. (10j), we get the final ELBO:
$$
\log p _ {\theta} (Y _ {T} ^ {0: t} | X _ {T} ^ {0: t}, \mathcal {C})
$$
(12a)
$$
\geq \mathbb {E} _ {q _ {\phi} (z ^ {G} | \mathcal {T})} \left[ \int_ {0} ^ {t} \mathbb {E} _ {q _ {\phi} (z ^ {t} | z ^ {G}, \mathcal {T} ^ {t})} \left[ \log p _ {\theta} (Y _ {\mathcal {T}} ^ {t} | X _ {\mathcal {T}} ^ {t}, z ^ {t}) \right] - D _ {\mathrm {K L}} \big (q _ {\phi} (z ^ {t} | z ^ {G}, \mathcal {T} ^ {t}) \| p _ {\theta} (z ^ {t} | z ^ {G}, \mathcal {C} ^ {t}) \big) \right]
$$
$$
- D _ {\mathrm {K L}} \left(q _ {\phi} \left(z ^ {G} | \mathcal {T}\right) \| p _ {\theta} \left(z ^ {G} | \mathcal {C}\right)\right) \quad (\text {B y})
$$
(By substitution)
(12b)
$$
= \mathbb {E} _ {q _ {\phi} (z ^ {G} | \mathcal {T})} \Big [ \int_ {0} ^ {t} \mathbb {E} _ {q _ {\phi} (z ^ {t} | z ^ {G}, \mathcal {T} ^ {t})} \Big [ \log p _ {\theta} (Y _ {\mathcal {T}} ^ {t} | X _ {\mathcal {T}} ^ {t}, z ^ {t}) \Big ] - D _ {\mathrm {K L}} \big (q _ {\phi} (z ^ {t} | z ^ {G}, \mathcal {T} ^ {t}) \| q _ {\phi} (z ^ {t} | z ^ {G}, \mathcal {C} ^ {t}) \big) \Big ]
$$
$$
- D _ {\mathrm {K L}} \left(q _ {\phi} \left(z ^ {G} | \mathcal {T}\right) \| q _ {\phi} \left(z ^ {G} | \mathcal {C}\right)\right), \tag {FinalELBO}
$$
(12c)
where the decoder  $p_{\theta}$  serves as the conditional prior network and is replaced by the encoder  $q_{\phi}$  serving as the surrogate posterior network.  $q_{\phi}$  can be seen to be producing two intermediate bottleneck distributions: (a)  $q_{\phi}(z^{G}|\mathcal{T})$  transforms inputs into a distribution over global latent variables, (b) conditioned on the global latent variables,  $q_{\phi}(z^t |z^G,\mathcal{T}^t)$  gathers the t-th task inputs and learns another distribution over the task-specific latent variables. The task-specific latent variables and their corresponding input covariates  $X^t$  are then used by the deterministic decoder  $p_{\theta}$  to decode their corresponding logit  $h_*$ . It is indeed this dependency of  $p_{\theta}$  on the task identifier  $t$  that makes inference a challenging task in real-world CL settings.
A.2 Single Task NPCL and its ELBO
Single-Task (ST) NPCL preserves all but the inter-task cross attention  $CA_{\mathrm{lat}}^{0:t}$  and the global distribution encoder  $\psi^G$  layers from the architecture of the NPCL (Sec. 4.3). The task-specific latent variables  $z^t$  are thus derived as:
$$
\left\{z _ {i} ^ {t} \right\} _ {i = 1} ^ {M} \sim \mathcal {N} \left(\psi_ {\mu} ^ {t} \left(s _ {i} ^ {t}\right), \psi_ {\sigma} ^ {t} \left(s _ {i} ^ {t}\right)\right) = \mathcal {N} \left(\mu_ {t}, \sigma_ {t} ^ {2}\right), \forall t \in T, \tag {13}
$$
where  $s_i^t$  and  $\psi^t$  carry the same meaning as in Eq. (16). For a fair comparison in Table 1, we fix  $M$  to be the same as the total number of global ancestral samples  $N$  in the NPCL. The corresponding
ELBO amounts to:
$$
\begin{array}{l} \log p _ {\theta} \left(Y _ {\mathcal {T}} ^ {0: t} \mid X _ {\mathcal {T}} ^ {0: t}, \mathcal {C}\right) (14a) \\ \geq \int_ {0} ^ {t} \mathbb {E} _ {q _ {\phi} \left(z ^ {t} \mid \mathcal {T} ^ {t}\right)} \left[ \log p _ {\theta} \left(Y _ {\mathcal {T}} ^ {t} \mid X _ {\mathcal {T}} ^ {t}, z ^ {t}\right) \right] - D _ {\mathrm {K L}} \left(q _ {\phi} \left(z ^ {t} \mid \mathcal {T} ^ {t}\right) \| p _ {\theta} \left(z ^ {t} \mid \mathcal {C} ^ {t}\right)\right) \quad (\text {D r o p p i n g} z ^ {G} \text {f r o m E q . (1 2 c)}) (14b) \\ \end{array}
$$
A.3 ELBO for the NP [14] and the ANP [27]
The NP [14] and the ANP [27] employ a single latent variable  $z^G$  to model the global correlation of all tasks. In particular, compared to Sec. 4.3, the task-specific self-attention layer  $SA_{\mathrm{lat}}^t$  and the task-specific distribution encoder  $\psi^t$  is no longer required. While this enables knowledge sharing among tasks, NPs and ANPs are limited in modeling finer intra-task stochastic factors. The ELBO can be given as:
$$
\begin{array}{l} \log p _ {\theta} \left(Y _ {\mathcal {T}} ^ {0: t} \mid X _ {\mathcal {T}}, \mathcal {C}\right) (15a) \\ \geq \mathbb {E} _ {q _ {\phi} (z ^ {G} | \mathcal {T})} \left[ \log p _ {\theta} \left(Y _ {\mathcal {T}} ^ {0: t} \mid X _ {\mathcal {T}}, z ^ {G}\right) \right] - D _ {\mathrm {K L}} \left(q _ {\phi} \left(z ^ {G} \mid \mathcal {T}\right) \| p _ {\theta} \left(z ^ {G} \mid \mathcal {C}\right)\right) \quad (\text {D r o p p i n g} z ^ {0: t} \text {f r o m E q . (1 2 c)}) (15b) \\ \end{array}
$$
where  $z^G$  is derived in a way similar to Eq. (16), and the inputs  $X_{\mathcal{T}}$  and  $\mathcal{C}$  belong to  $[0,t]$  tasks without relying on the task labels for being encoded.
B NPs for Meta-Learning (ML) vs Continual Learning (CL)
Resemblance. For both ML and CL settings, we have multiple tasks and would like to learn an NP with flexible conditioning that generates task-specific functions. Specifically, for each task  $t$ , we have  $Y^{t} = F_{[\phi ;\theta ]}^{t}(X^{t};z^{t})$  where the latent variable  $z^t$  is conditioned on the task-specific context  $C^t$ . As  $z^t$  is task-specific, this, in turn, makes  $F^t$  task-specific.
Differences. There are two major differences between the conventional NP for ML and our proposed NP for CL (NPCL):
1. Architectural difference. In standard NPs,  $z^t$  is conditioned only on the task-specific context  $\mathcal{C}^t$  while in the NPCL,  $z^t$  is conditioned on  $[\mathcal{C}^t; z^G]$  where  $z^G$  is the global latent derived from the global context. The added conditioning of global latent reflects the need for cross-task knowledge transfer in CL and thus endows a two-level hierarchy into our model.
2. Functional difference. The functional difference between the ML and CL (inference) settings call for another adaptation in the NPCL. Namely, ML aims to learn NPs that learn  $F_{[\phi ;\theta ]}$  that can generalize to new tasks. That is why NPs are tested on a new task  $t_*$  given its context  $C_*$ . On the other hand, in CL, we wish to learn an  $F$  that can perform well on all the seen tasks as there is no new task during inference. Given the absence of a labeled context during inference, our test-time context samples are thus a subset of the training data from the tasks seen so far.
In a nutshell, CL calls for learning a general function that is task-specific but also performs well on all the seen tasks subject to a limited rehearsal memory. To achieve this, we leverage an NP's ability of generating task-specific functions with flexible conditioning. But instead of using the conventional NPs for ML, we propose a hierarchical model to introduce more inter-task knowledge sharing, thus tailored for the CL problem.
C Further on the NPCL Architecture
In the following, we denote multi-head dot product self-attention [48] by  $SA(K,V,Q)$  where  $K$ ,  $V$ , and  $Q$  are the keys, values and queries, respectively. The equivalent notation for cross-attention is  $CA(K,V,Q)$ .
Latent Encoder. The latent path learns the functional prior and posterior from the context and the target sets, respectively. Each label-concatenated input is projected as  $\Phi_i^{\mathrm{lat}} = \mathrm{MLP}([x_i; y_i])$ ; then
subjected to two attention operations. First, per-task projections form the keys, values, and queries to taskwise self-attention layers  $SA_{\mathrm{lat}}^t (\Phi_i^{\mathrm{lat}}, \Phi_i^{\mathrm{lat}}, \Phi_i^{\mathrm{lat}}): \Phi_i^{\mathrm{lat}} \to s_i^t$  that produce order-invariant encodings  $s_i^t$  over the task  $t$ . Second, all encodings  $\{s_i^{0:t}\}_{i=1}^{n+m}$  serve as the keys, values, and queries to the cross-attention layers  $CA_{\mathrm{lat}}^{0:t}(s_i^t, s_i^t, s_i^t): s_i^t \to s_i^G$  that enrich their order-invariance from intra-task  $s^t$  to inter-task  $s^G$ .  $s^t$  and  $s^G$  are then used to derive the global  $z^G$  and the task-specific latent variables  $z^t$ .
Such globally attended inputs are passed in parallel to two MLP layers constituting the global distribution encoder  $\psi^G$  whose outputs together parameterize the global distribution  $\mathcal{N}(\mu_G, \sigma_G^2)$  over the input set, i.e.,  $\psi^G(s^G): \{s_i^G\}_{i=1}^{n+m} \to (\mu_G, \sigma_G^2)$ . Samples  $\{z_i^G\}_{i=1}^N$  drawn from this distribution are proxies for the variables capturing the global correlation over all tasks in the input set. It is indeed this sampling step that induces the stochasticity into the learned posteriors of the NPCL.
To model finer task-specific distribution for task  $t$  conditioned on the global distribution, we retain the task-specific self-attended representations  $s_i^t$  and concatenate these with the global latent variables  $\{z_i^G\}_{i=1}^N$  to produce  $N$  distinct encodings per input point. These encodings are then passed through the t-th task distribution encoder  $\psi^t$  that again constitutes a mean and a variance MLP head and produces outputs that parameterize the t-th task distribution  $\mathcal{N}(\mu_t, \sigma_t^2)$ , i.e.,  $\psi^t(s^t): \{s_i^t\}_{i=1}^{n+m} \to (\mu_t, \sigma_t^2)$ . Samples  $\{z_j^T\}_{i=j}^M$  drawn from each such distribution thus capture the per-task stochastic factors. To limit the randomness in the learned prior/posterior, we use  $M = 1$ . The latent encoder thus outputs a subtotal of  $N \times (t + 1)$  encodings per input point.
Put together, the global and task-specific latent variables can be derived as:
$$
\begin{array}{l} \left\{z _ {i} ^ {G} \right\} _ {i = 1} ^ {N} \sim \mathcal {N} \left(\psi_ {\mu} ^ {G} \left(s ^ {G}\right), \psi_ {\sigma} ^ {G} \left(s ^ {G}\right)\right) = \mathcal {N} \left(\mu_ {G}, \sigma_ {G} ^ {2}\right), \tag {16} \\ \{z _ {i} ^ {t} \} _ {i = 1} ^ {M} \sim \mathcal {N} (\psi_ {\mu} ^ {t} (s _ {i} ^ {t}, z _ {i} ^ {G}), \psi_ {\sigma} ^ {t} (s _ {i} ^ {t}, z _ {i} ^ {G})) = \mathcal {N} (\mu_ {t}, \sigma_ {t} ^ {2}), \forall t \in T, \\ \end{array}
$$
where  $\psi^G$  and  $\psi^t$  are the global and per-task distribution encoders, respectively.
Deterministic Encoder. The deterministic path is similar to that of an ANP [27] where the context projections  $\Phi_i^{\mathrm{det}} = \mathrm{MLP}([x_i;y_i])$  form the keys, queries and values for a self-attention operation,  $SA_{\mathrm{det}}(\Phi_i^{\mathrm{det}},\Phi_i^{\mathrm{det}},\Phi_i^{\mathrm{det}}):\Phi_i^{\mathrm{det}}\to r_i$ . The resulting order-invariant context representations  $\{r_i\}_{i = 1}^m$  are fed as values to a subsequent target-to-context cross-attention operation  $CA_{\mathrm{det}}$ . The keys  $x_{i}$  and queries  $x_{*}$  for  $CA_{\mathrm{det}}$  come from the context  $x_{i}\in X_{\mathcal{C}}$  and target  $x_{*}\in X_{\mathcal{T}}$  covariates, respectively, i.e.,  $CA_{\mathrm{det}}(x_i,s_C,x_*):x_*\rightarrow r_*$  where  $r_*$  is invariant to the order of context.
Decoder. Different from other NP variants, the NPCL decoder adopts separate decoding mechanisms during training and inference. At train time, we use the available task identity to filter the true  $N$  out of  $N*(t+1)$  latent path outputs to be processed by the decoder. After this, the decoder concatenates a target input  $x_{*}^{t} \in X_{\mathcal{T}}^{t}$  with its  $N$  true task-specific latent variables  $\{z_{i}^{t}\}_{i=1}^{N}$  obtained from the latent path and its order-invariant feature  $r_{*}$  obtained from the deterministic path thus resulting in  $N$  distinct inputs. For  $N > 1$  samples of  $z_{t}$ , we first make  $N$  copies of  $x_{*}$  and  $r_{*}$  each, and then concatenate these with each  $z^{t}$ .  $p_{\theta}$  thus performs the projection  $p_{\theta}([x_{*}; r_{*}; \{z_{i}^{t}\}_{i=1}^{N}]) : x \to h_{*}$  where  $x \in \mathbb{R}^{f+2*o}$  and  $h_{*}$  are the logits of an MLP classifier for the target label  $y_{*}$ . We detail the inference-time decoding in Sec. 4.5.
D Experiments and Reproducibility
Configuration. For a fair comparison with the benchmarks of Buzzega et al. [4], we fix the batch sizes for new task's samples and for replay samples to 32 each for the class-IL datasets and to 128 each for the domain-IL datasets. Both the context and target datasets use the same set of augmentations. For S-CIFAR-10, S-CIFAR-100, and S-Tiny-ImageNet, we apply random crops and horizontal flips to both stream and buffer examples following Buzzega et al. [4] and Boschini et al. [3]. For each setting of memory size on each dataset, the NPCL adopts the same learning rate (LR) as reported in Buzzega et al. [4] and Boschini et al. [3]. However, the NPCL training additionally relies on linearly increasing the learning rate (LR) over a period of 4000 iterations for class-IL and 40 iterations for domain-IL settings. We further apply gradient clipping [40] on L2-norm of the NPCL parameters with a cap of 10000.
Hyperparameter tuning. We arrive at the best hyperparameter settings for each of our datasets through grid search over a validation set made of  $10\%$  of the training set on each dataset. The search range for number of samples  $N$  from the global distribution  $\mathcal{N}(\mu_G, \sigma_G^2)$  is [2, 5, 10, 20, 50, 100]. Out of these, we found  $N = 50$  during training and  $N = 10$  during evaluation to perform better in general across all settings.
Similarly, we conducted a grid search over the batch size of the context set  $\mathcal{C}$  over the range  $[1/16, 1/8, 1/4, 1/2, 1, 1.25]$  of the original (target) batch sizes for each of the dataset. In general, we found that fixing the context batch size to  $1/8$  of the target batch size performed better across all datasets. Such context batches are sampled from a context dataset  $\mathcal{D}_{\mathcal{C}}^{t}$  for each task  $t$ .  $\mathcal{D}_{\mathcal{C}}^{t}$  is itself created by randomly selecting a subset of the training samples for each class at the beginning of each incremental training task. To decide on the size of the subset for each class, we ran a grid search over the range  $[50, 100, 150, 200]$  samples per class and found that incorporating 100 random samples per class into  $\mathcal{D}_{\mathcal{C}}^{t}$  performed well across all datasets.
Finally, to decide on the loss weights  $\alpha$ ,  $\beta$ ,  $\gamma$  and  $\delta$  for  $D^t$ ,  $D^G$ ,  $\mathcal{L}_{\mathrm{GR}}$ , and  $\mathcal{L}_{\mathrm{TR}}^t$ , we ran gridsearch for each over possible values [0.0, 0.01, 0.05, 0.08, 0.1, 0.15, 0.2, 0.4]. We report the best settings across datasets in Table 9:
S-CIFAR-10	S-CIFAR-100	S-Tiny/ImageNet	P-MNIST	R-MNIST
α	0.05	0.05	0.01	0.1	0.1
β	0.01	0.01	0.01	0.05	0.05
γ	0.2	0.08	0.05	0.1	0.1
δ	0.1	0.1	0.1	0.15	0.15
Table 9: Hyperparameters for loss contributions that were tuned on validation sets for each dataset.
E On the number of Monte Carlo samples
Figure 6 shows the effect of the number of Monte Carlo (MC) samples for global  $N$  and task-specific  $M$  latent variables on accuracy during inference. In particular, we observe two favorable spots in terms of accuracy, one centered around  $(M = 1, N = 50)$  and the other around  $(M = 10, N = 20)$ . It is worth noting that the total number of inference time MC samples grow quadratically with the number of tasks  $t$ , i.e.,  $\mathcal{O}(NMt)$ , and that a higher number of samples leads to a larger computational overhead. For instance, based on Eq. (9) in the main paper, the inference on the 10-th task of S-CIFAR-100, i.e.,  $t = 10$ , given the two favorable spots amounts to selecting the set of task-specific module predictions with the least uncertainty from a total of (a)  $1 \times 50 \times 10 = 500$  predictions using  $(M = 1, N = 50)$ , and (b)  $10 \times 20 \times 10 = 2000$  predictions using  $(M = 10, N = 20)$ . We, therefore, opt for the more efficient setting of  $(M = 1, N = 50)$  throughout our experiments in the paper.
Figure 6: The effect of the number of Monte Carlo (MC) samples of global (N) and task-specific (M) latent variables on S-CIFAR-100 ( $\mathcal{M}_{size} = 500$ ) accuracy.
Method	S-CIFAR-10
Class-IL	P-MNIST
Domain-IL	R-MNIST
Domain-IL
oEWC	-91.64	-36.69	-24.59
SI	-95.78	-27.91	-22.91
LwF	-96.69	-	-
Msize	200	500	200	500	200	500
ER	-61.24	-45.35	-22.54	-14.90	-8.24	-7.52
GEM	-82.61	-74.31	-29.38	-18.76	-11.51	-7.19
A-GEM	-95.73	-94.01	-31.69	-28.53	-19.32	-19.36
iCaRL	-28.72	-25.71	-	-	-	-
FDR	-86.40	-85.62	-20.62	-12.80	-13.31	-6.70
GSS	-75.25	-62.88	-47.85	-23.68	-20.19	-17.45
HAL	-69.11	-62.21	-15.24	-11.58	-11.71	-6.78
DER	-40.76	-26.74	-13.79	-8.04	-5.99	-3.41
ANP	-62.80	-49.18	-28.79	-16.44	-12.08	-10.63
ST-NPCL	-46.91	-32.50	-17.03	-12.40	-7.9	-8.11
NPCL (ours)	-39.11	-27.62	-12.81	-8.60	-5.70	-4.10
Table 10: Backward transfer scores for the experiments in Table 1. Best results are in red. Second best results are in blue. All runs of the ANP, the ST-NPCL and the NPCL in the CL settings rely on experience replay (ER).
F Results: Backward Transfer
Table 10 reports the backward transfer for the accuracy scores mentioned in table 1. We further compute the backward transfer based on uncertainty scores to study the effect of forgetting on uncertainty. Fig. 7 shows the correlation between backward transfer of accuracy and uncertainty for the domain-IL datasets P-MNIST and R-MNIST.
Figure 7: Backward transfer scores of tasks based on accuracy and uncertainty on domain-IL datasets with  $|\mathcal{M}| = 500$ : a higher negative backward transfer on accuracy correlates with a higher positive backward transfer on uncertainty and vice-versa. For better visibility, the uncertainty-based backward transfer scores have been scaled by a factor of 100.
G.1 On the effect of regularization on the learned distributions
We record the per epoch L1-norms of global and task-specific means and variances on the last incremental task (task 4) of S-CIFAR-10. As shown in Fig. 8(a) and Fig. 8(b), regularizing the global distribution (GR) alleviates forgetting by limiting the learning of the global and the current task's (task 4) means and variances. This is evident through larger L1-norm of means and smaller L1-norm of variances when  $\mathrm{GR} = 0$ , i.e., +TR setting. On the other hand, excluding all the objectives, i.e., the Baseline NPCL as well as excluding TR from the learning objectives, i.e., +GR setting lead to
relatively unstable evolution of the past task means and variances, hence characterizing an increased forgetting. Including both GR and TR in the objective, i.e., the NPCL helps find a balance between preserving the global and the past-task distributions while facilitating the learning of the current task distribution.
(a) Effect on the global and task-specific means.
Variances
(b) Effect on the global and task-specific variances.
Figure 8: Effect of the proposed global (GR) and task-specific (TR) regularizations on the learning of global and task-specific means and variances during the last incremental training task (task 4) of S-CIFAR-10. The NPCL uses both GR and TR while the baseline NPCL uses neither of them.
G.2 How does forgetting effect uncertainty?
Fig. 9 ablates the average accuracies and uncertainties of each task head predictions over the test set of each task at the end of incremental training on S-CIFAR-100. Similar to S-CIFAR-10 (Fig. 4), we observe that the accuracy of predictions made by true task heads are higher than the rest. For predictive uncertainties, the trend is the opposite. Also, more recently trained tasks show lesser forgetting both in terms of accuracy (higher values) and uncertainty (lower values). This generalizes
our conclusion on S-CIFAR-10 regarding the outreach of forgetting in CL going beyond accuracy and to other aspects of learning such as the model's predictive confidence.
(a) Accuracy heatmap
(b) Uncertainty heatmap
Figure 9: Heatmaps depicting the average accuracy and uncertainty of individual task test sets per task head on S-CIFAR-100 with  $|\mathcal{M}| = 500$  over an individual run.
G.3 On the storage gain of the NPCL over DER
Table 11 compares the total episodic memory sizes of the NPCL (ours) and DER [4]. We report storage sizes as the dimension of a single 1-d vector constructed by flattening all the vectors that need to be stored by each method in the episodic memory. Namely, the NPCL stores  $2*t + 2$  vectors of fixed dimension  $\mathbb{R}^{|o|}$  where  $t$  is the total number of tasks in a dataset and  $o$  is the output size of the mean and variance heads. On the other hand, DER stores  $|\mathcal{M}|$  number of logits of dimension  $\mathbb{R}^{|N_C|}$  where  $N_{C}$  denotes the total number of classes in a CL dataset. As a result, the NPCL has significant storage gains on settings with either large number of classes or a larger memory size. It is worth noting that both the NPCL and DER rely on storing original input images and therefore, our comparison does not take the inputs into account.
Method	S-CIFAR-10	S-CIFAR-100	S-Tiny/ImageNet	P-MNIST	R-MNIST
\( {\mathcal{M}}_{\text{size }} \)	200	500	500	2000	200	500	200	500	200	500
DER [4]	2000	5000	50000	200000	40000	100000	2000	5000	2000	5000
NPCL (ours)	3272	3572	6132	7632	5832	6132	1544	1844	1544	1844
Storage gain (%)	-63.6	28.56	87.746	96.184	85.42	93.868	22.8	63.12	22.8	63.12
Table 11: Storage size comparison of the NPCL with DER across different experimental settings of Table 1. Gains are marked in bold.
G.4 On the importance of correct context
To study the significance of correct task-specific context  $\mathcal{C}^t$  for the NPCL, we design a simple experiment. While training on an incremental task  $t > 0$  onwards, after having derived the global latent samples  $\{z^G\}_{i=1}^N$ , we tinker with the flow of the task-specific context points  $\mathcal{C}^t$  to the different task-specific encoders of the NPCL. In specific, instead of directing  $\mathcal{C}^t$  to the  $t$ -th encoder, we misdirect it to the task encoder  $j \ni j \neq t$  where,  $j$  is chosen at random from the pool of all seen task ids. Note that the presence of such randomly allocated context points during training implies that we now have a noisy task-specific prior  $q_{\phi}(z^t | z^G, \mathcal{C}^j)$  to match in the ELBO, i.e., a corrupted second term on the right-hand side of eq. (5). We keep all our other training settings (including the loss coefficient values) unchanged.
Table G.4 compares the performance of the NPCL with the noisy task-specific priors on the two different memory sizes of S-CIFAR-10. While the performance gain of the NPCL over its noisy prior counterpart remains significant, we observe that in comparison with the ST-NPCL (which lacks hierarchy), the presence of noisy priors degrades the performance of the NPCL (which has hierarchy) further as the replay memory size increases from 200 to 500. This is because, with a larger memory size, more context points from past tasks are diverted to the random task components during training. This leads to a noisier task-specific prior matching. Such noisy priors further lead to higher