SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Datasets. We evaluate the NPCL on class and domain incremental learning (IL) settings. For class-IL, we use three public datasets: sequential CIFAR10 (S-CIFAR-10) [35], sequential CIFAR100 (S-CIFAR-100) [54], and sequential Tiny ImageNet (S-Tiny-ImageNet) [6]. For domain-IL, we use Permuted MNIST (P-MNIST) [30] and Rotated MNIST (R-MNIST) [35]. S-CIFAR-10, S-CIFAR100, and S-Tiny-ImageNet host 10, 100, and 200 classes each with 5000, 500, and 500 training images and 1000, 100, and 50 test images per class, respectively. The number of sequential tasks for S-CIFAR-10 is 5 (2 classes per task); for S-CIFAR-100 and S-Tiny-ImageNet is 10 (10 and 20 classes per task, respectively); for P/R-MNIST is 20. P-MNIST creates tasks out of MNIST [32] by randomly permuting the pixels, and R-MNIST does it by rotating images randomly in  $[0,\pi)$ .
Architectures. For a fair comparison against other methods, we rely on the Mammoth CL benchmark [3]. Our backbone for class-IL experiments is a ResNet-18 [20] without pretraining, while for domain-IL, we rely on a fully connected (FC) network with two hidden layers [35]. The NPCL relies on Xavier initialized [15] FC layers with two 256-d hidden layers for class-IL and one 32-d layer for domain-IL setups. For class-IL, each FC layer is followed by layer normalization [1] and ReLU.
Configuration and hyperparameters. We train all models using SGD optimizer. The number of training epochs per task for S-Tiny-ImageNet is 100, for S-CIFAR-(10/100) is 50, and that for (P/R)-MNIST is 1. We detail further the configurations, hyperparameters, and their tuning in App. D.
Method	S-CIFAR-10
Class-IL	S-CIFAR-100
Class-IL	S-Tiny-ImageNet
Class-IL	P-MNIST
Domain-IL	R-MNIST
Domain-IL
Joint ResNet	92.2 ± 0.15	70.44	59.99 ± 0.19	94.33 ± 0.17	95.76 ± 0.04
Joint NP	91.66 ± 0.11	70.58 ± 0.24	59.83 ± 0.17	95.02 ± 0.21	95.37 ± 0.07
Joint ANP	91.26 ± 0.16	70.77 ± 0.21	60.14 ± 0.17	95.39 ± 0.18	95.85 ± 0.05
Joint NPCL	92.74 ± 0.12	71.46 ± 0.20	60.18 ± 0.22	95.97 ± 0.14	96.11 ± 0.03
Multitask NPCL	69.15 ± 0.09	53.6 ± 0.21	35.53 ± 0.13	87.40 ± 0.10	89.21 ± 0.02
oEWC [44]	19.49 ± 0.12	-	7.58 ± 0.10	75.79 ± 2.25	77.35 ± 5.77
SI [54]	19.48 ± 0.17	-	6.58 ± 0.31	65.86 ± 1.57	71.91 ± 5.83
LwF [34]	19.61 ± 0.05	-	8.46 ± 0.22	-	-
M_size	200	500	500	2000	200	500	200	500	200	500
ER [43]	44.79 ± 1.86	57.74 ± 0.27	22.10	38.58	8.49 ± 0.16	9.99 ± 0.29	72.37 ± 0.87	80.6 ± 0.86	85.01 ± 1.90	88.91 ± 1.44
iCARL [42]	49.02 ± 3.20	47.55 ± 3.95	46.52	49.82	7.53 ± 0.79	9.38 ± 1.53	-	-	-	-
FDR [2]	30.91 ± 2.74	28.71 ± 3.23	-	-	8.70 ± 0.19	10.54 ± 0.21	74.77 ± 0.83	83.18 ± 0.53	85.22 ± 3.35	89.67 ± 1.63
RPC [41]	-	-	22.34	38.33	-	-	-	-	-	-
DER [4]	61.93 ± 1.79	70.51 ± 1.67	36.6	51.89	11.87 ± 0.78	17.75 ± 1.14	81.74 ± 1.07	87.29 ± 0.46	90.04 ± 2.61	92.24 ± 1.12
NP [14]	46.1 ± 3.44	59.3 ± 2.76	22.92	38.70	8.32 ± 0.62	10.2 ± 0.34	70.02 ± 1.44	79.44 ± 0.81	85.03 ± 2.7	88.16 ± 1.66
ANP [27]	46.67 ± 1.23	58.77 ± 0.65	23.2	39.06	8.81 ± 0.93	9.75 ± 0.90	73.55 ± 0.66	80.98 ± 0.57	85.70 ± 1.39	89.21 ± 0.93
ST-NPCL (w/ only per-task latent)	54.6 ± 2.14	65.22 ± 1.89	28.45	42.1	10.92 ± 1.03	13.7 ± 1.35	76.4 ± 1.62	82.06 ± 0.92	86.99 ± 3.07	89.64 ± 2.11
Naive NPCL (w/o task head inf.)	19.54 ± 3.44	20.71 ± 3.09	18.27	18.90	7.19 ± 1.02	8.48 ± 0.90	68.37 ± 1.58	73.3 ± 0.81	81.13 ± 2.91	83.69 ± 2.24
NPCL (ours)	63.78 ± 1.70	71.34 ± 1.48	37.43	46.71	12.44 ± 0.59	15.29 ± 1.02	83.11 ± 0.90	86.52 ± 0.77	91.48 ± 1.79	92.07 ± 1.39
Baselines. We employ several CL methods to compare the NPCL with. Regularization-based methods include oEWC [44] and SI [54]; knowledge distillation-based methods include iCaRL [42] and LwF [34]; rehearsal-based methods are ER [43], RPC [41], FDR [2], DER [4]. Among neural processes, we use the NP [14], the ANP [27], and the Single Task (ST) NPCL (see App. A.2) with only per-task latent variables. We use five non-CL benchmarks as upper bounds on the performances: Joint ResNet / NP / ANP / NPCL perform joint training of all tasks using a single task head while the multitask NPCL infers task heads in joint training using Eq. (9). Finally, the naive NPCL inference averages the logits of all task heads.
Table 1 reports the average accuracy after training on all tasks. Across all settings, the NPCL boosts the performance of the ER and achieves either comparable or better results against the state-of-the-art (SOTA), e.g., DER. Compared to the regularization-based oEWC and SI, the NPCL obtains a significant gain in performance. This is because the former methods calculate weight importance, which is liable to changes with new tasks. Regularizing explicitly towards the global and per-task distributions of past tasks helps the NPCL overcome this. Further, on both class and domain-IL, the NPCL stands out in the most challenging setting where the episodic memory size is the smallest. On domain-IL where the shift occurs within the domain instead of the classes, the performance of a number of methods degrade as they forget the relations among a task's classes. Preserving the tasks' distributions helps the NPCL maintain valuable information in this case. Analyzing the backward transfer (BWT) scores [39] shows that the NPCL's forgetting is competitive or lesser than the SOTA (see Table 10). Lastly, we note that the ST-NPCL with no hierarchy lags in BWT and accuracy due to limited knowledge transfer between tasks.
Why uncertainty-aware inference works? For our uncertainty-aware task head inference mechanism to be effective, a CL model must produce probabilities that align well with the ground truth labels of the test samples. We thus ablate the calibration errors for different CL baselines using the well-established Expected Calibration Error (ECE) [16] and Adaptive Calibration Error (ACE) [38] metrics. Table 2 shows that the NPCL has the least calibration error across S-CIFAR-10 ( $\mathcal{M}_{\mathrm{size}} = 200$ ) and S-CIFAR-100 ( $\mathcal{M}_{\mathrm{size}} = 500$ ). In general, the probabilistic nature of the ANP [27] and
Table 1: Classification accuracy for standard CL benchmarks across 10 runs. The best results are in red. The second best results are in blue. All runs of NP variants in the CL settings rely on ER. S-CIFAR-100 results are from Boschini et al. [3] while the rest are taken from Buzzega et al. [4].
Method	S-CIFAR-10	S-CIFAR-100
Metric	ECE	ACE	ECE	ACE
ER [43]	0.4553	0.8532	0.6459	0.9499
DER [4]	0.2991	0.8391	0.2484	0.9447
ANP [27]	0.34	0.8495	0.5441	0.9477
NPCL (ours)	0.2103	0.8155	0.1995	0.9421
Table 2: Model calibration errors averaged across 10 runs.
(a) Accuracy
(b) Uncertainty
Figure 4: Heatmaps depicting the taskwise averaged accuracy and uncertainty of test samples per task head on S-CIFAR-10.
(a) Accuracy
Figure 5: Effect of context set size  $(\mathcal{M}_{size} \in \{5, 50, 100, 200\})$  on the accuracy and uncertainty of the NPCL on S-CIFAR-10.
(b) Uncertainty
the NPCL benefits them in confidence calibration over the deterministic methods with comparable accuracies, i.e., ER [43] and DER [4].
Uncertainty-Accuracy trade-off. Fig. 4 ablates the average accuracies and uncertainties of each task head predictions over the test set of each task on S-CIFAR-10 (see App. G.2 for S-CIFAR-100). First, we observe that the accuracy of predictions made by true task heads are, in general, a magnitude higher than the rest. For uncertainty, this trend is reversed. This verifies our assumption that restricting latent heads to learn only their true label distribution makes them more confident in modeling the within-task samples. Second, for recently trained tasks, the uncertainty differences between the true task heads and the rest are greater than the earlier tasks. This, in general, suggests that the extent of forgetting goes beyond a CL model's accuracy and to other aspects of its learning such as its predictive confidence. To support the latter claim, we probe the BWT of uncertainty and see a strong correlation with the BWT of accuracy (see Fig. 7).
Learning objectives. Table 3 shows the impact of distribution regularization, with the baseline being the NPCL trained with no regularization. We observe that the baseline performs worse than the ER as the NPCL layers forget more. Including TR in our objectives leads to the single-most gain over the baseline. We further study how these objectives guide the learning of the global and task-specific distributions with training (see App. G.1). We observe that the NPCL w/ TR leads to better learning of the current task as well as preserving the
Method	S-CIFAR-10	S-Tiny-ImageNet
ER [43]	44.79	8.49
Baseline (w/o GR or TR)	32.24	7.15
NPCL (w/ only GR)	50.68	8.61
NPCL (w/ only TR)	57.28	11.36
NPCL (w/ GR and TR)	63.78	12.44
Table 3: Accuracy w/ learning objectives
past task distributions but at the cost of drifting the global distribution. The NPCL w/ GR restricts the drift of the global distribution but not for the per-task distributions. The NPCL w/ GR and TR strikes a balance in between.
Effect of Monte Carlo (MC) samples. We spot two combinations of the number of global  $N$  and task-specific  $M$  MC samples in favor of performance. Out of these, we choose the one with the superior computational efficiency (see App. E for details).
Context size. We study the average accuracy (Fig. 5(a)) and uncertainty (Fig. 5(b)) after training on S-CIFAR-10 with  $|\mathcal{M}| = 200$ , and then varying the context sizes during inference. Similar to other NPs [49, 12], we find a positive correlation between context size and performance, indicating that the NPCL utilizes useful information from diverse contexts, thereby reducing its task inference ambiguity.
Few-shot replay settings. A key strength of NPs remains their few-shot learning capability. To study how well the NPCL retains this trait against other CL baselines, we ablate their accuracy and ECE [16] on rehearsal memory sizes of 5 and 10 (see Table 4). We find that on both memory sizes, the NPCL outperforms ER [43] and DER [4]. For  $\mathcal{M}_{size} = 5$  on S-CIFAR-100, we observe that the ER outperforms the DER in terms of accuracy. However, the latter still offers more confident predictions (characterized by a lower ECE). This implies that on few-shot CL replay settings while regularizing the predicted logits towards their old forms – as done by the DER – helps improve the predictive confidence over the ER, regularizing the task distributions towards their old forms – as done by the NPCL – remains the superior way to enhance the model's predictive confidence.
Method	S-CIFAR-10	S-CIFAR-100
\( \mathcal{M}_{\text{size}} \)	5	10	5	10
Metric	Acc.	ECE	Acc.	ECE	Acc.	ECE	Acc.	ECE
ER [43]	22.11	0.7281	25.39	0.696	9.44	0.8003	9.69	0.8014
DER [4]	21.05	0.5931	25.2	0.5107	8.96	0.4593	10.97	0.542
NPCL (ours)	22.98	0.4709	26.15	0.441	10.22	0.39	12.64	0.4717
Storage efficiency. For each task, the NPCL stores two new vectors - task-specific mean and variance, and replaces the global mean and variance with the current global ones. The NPCL storage thus scales constantly in the size  $|\mathcal{M}|$  of the memory. This offers a strong edge on storage efficiency when compared to DER [4] scaling quadratically, i.e.,  $|\mathcal{M}| \times N_C$  where  $N_C$  is the total number of classes. For instance, on S-Tiny-ImageNet with  $|\mathcal{M}| = 500$ ,  $N_C = 200$ , the NPCL's cumulative storage amounts to a (flattened) vector of size 6132 ( $256 \times 10 \times 2$  for 256-d means and variances of 10 tasks plus  $256 \times 2$  for 256-d global mean and variance plus 500 for 1-d task labels) while that of DER amounts to 100,000 ( $200 \times 500$  for logits of 500 memory samples), i.e., a  $93.868\%$  relative storage efficiency. We report the storage efficiency of the NPCL over DER across all settings in App. G.3.
5.3 Applications of Uncertainty Quantification
The probabilistic nature of the NPCL offers it an edge in leveraging data-driven UQMs. To further study the usage of predictive uncertainties, we conduct two experiments with a trained NPCL model.
Novel data identification. Novel data identification seeks to distinguish out-of-distribution data  $(\mathcal{D}_{\mathrm{OOD}})$  from in-domain data  $(\mathcal{D}_{\mathrm{ID}})$ . Forgetting makes CL models struggle further on the task [19]. The probabilistic sampling in the NPCL opens the door for leveraging its predictive variances - which are more reliable estimates of aleatoric uncertainty than pointwise predictions [22]. For the  $N$  predicted logits, we thus compute the variances over their softmax scores,  $\sigma^2 (\delta (h_*))$ , and over their uncertainty scores,  $\sigma^2 (U(h_*))$ . Table 5 evaluates these met-
Table 4: Few-shot replay results: Accuracy (Acc.) and ECE [16] of different methods with very small buffer sizes of 5 and 10 for S-CIFAR-10 and S-CIFAR-100 averaged across 3 runs.
Incremental step	ID=CIFAR-10,OOD=CIFAR-100
ID(δ)	OOD(δ)	ID(H)	OOD(H)
1	1e-6	1e-5	9.3e-6	8.4e-5
2	2.6e-6	1.4e-5	6.3e-5	2.2e-4
3	2.3e-6	6.2e-6	6.7e-5	2.1e-4
4	8.1e-7	4.8e-6	4.6e-5	2.2e-4
5	7.1e-7	1.7e-6	4.6e-5	1.1e-4
rics for ID (S-CIFAR-10) and OOD (first 10 classes of S-CIFAR-100) data after each task. We observe that the variance scores of either metrics on  $\mathcal{D}_{\mathrm{ID}}$  are up to a magnitude lower than those on  $\mathcal{D}_{\mathrm{OOD}}$ . We further observe an overall decrease in the variances with the arrival of further incremental tasks. This could be attributed to the generalization of more low-level features in the novel data as in-domain [18, 16]. We detail further novel data identification experiments in App. G.5.
Instance-level model confidence evaluation. The confidence evaluation framework of Han et al. [17] provides finer granularity for assessing the predictive confidence of classification models (see App. G.6 for more details and normality test). Table 6 shows the results of one run of the framework after training on S-CIFAR-10. Here, we use the task identity to select the latent head per class. We observe the mean prediction interval width (PIW) of the true class label among the correct predictions to be narrower than that of the incorrect predictions, implying that the NPCL's variations of predicted class labels are smaller when the predictions are correct. We also notice a higher accuracy among the test instances rejected by the  $t$ -test than those not rejected.
Table 5: Average variances over softmax  $(\delta)$  and entropy  $(H)$  scores on in-domain and out-of-domain test sets using  $N = 50$  ancestral samples.
Class	Accuracy	PIW	Accuracy by t-test status
Correct	Incorrect	Rejected	Not Rejected
1	82.30	74.17	102.21	83.37	50.00
2	94.00	62.90	79.86	94.07	80.00
3	74.00	54.92	68.48	74.14	64.29
4	71.50	65.42	74.32	72.06	25.00
5	84.80	92.93	106.90	85.37	22.22
6	76.50	75.22	103.58	76.58	60.00
7	94.20	104.9	129.56	94.39	3.00
8	90.50	81.10	127.06	91.12	22.22
9	96.90	72.81	110.86	97.00	66.67
10	96.30	80.60	109.56	96.48	60.00
Table 6: PIW (multiplied by 100) and  $t$ -test results for the first three classes of S-CIFAR-10 inferred from their respective task heads.
We list the key limitations of the NPCL to facilitate future research directions.
Incompetence of dot-product attention. Similar to the ANP [27], the NPCL employs the permutation-invariant scaled-dot product attention [48] to weigh the relevant context and target embeddings. Visualizing the attention weights computed by the cross-attention layers of the deterministic path shows us that the top attended context for the target queries often contain points belonging to other CL tasks (Fig. 11(a)). This limits the performance sensitivity of the NPCL with respect to the increase in context thus resulting in a lag of accuracy behind SOTA on CL setups with larger episodic memory sizes (see Table 1). To further verify the relevance of the attended context, we visualize the self-attention weights of all context points. Fig. 11(b) shows that the lowest or the maximum values in the context dataset have larger weights. Such an observation is in line with existing works pointing that the scaled-dot product attention can derive irrelevant set encodings of the context points and can thus lag at exploiting the context embeddings properly [28].
Computational overhead. Table 7 compares the number of parameters of the NPCL with ER / DER [3] where the latter relies solely on the ResNet-18 backbone as they do not exploit parameter isolation for task heads. Overall, the percentage increase in parameter number is  $57.6\%$  for S-CIFAR-10,  $46.57\%$  for S-CIFAR-100 and S-Tiny/ImageNet, and  $55.25\%$  for P/R-MNIST.
Method / Dataset	S-CIFAR-10	S-CIFAR-100	S-Tiny-ImageNet	P/R-MNIST
ER / DER [3]	11,173,962	11,220,132	11,220,132	89,610
NPCL	19,397,706	24,091,556	24,091,556	162,166
Inference time complexity. The reliance on self-attention means that the inference time complexity of the NPCL is  $\mathcal{O}(n*m)$ , where  $n$  is the number of context points (sampled from the episodic memory) and  $m$  is the number of target points (the number of test samples). Due to this, the runtime for inference scales polynomially with the number of context points (sampled from the buffer). Table 6 reports the runtime of the NPCL on S-CIFAR-10 and S-CIFAR-100 settings by varying the context sizes. For reference, the first row reports the runtime of ER / DER whose inference complexity is  $\mathcal{O}(1)$  in the memory but
Table 7: Comparison of the total number of parameters for ER / DER against the NPCL.
Method	S-CIFAR-10
ER / DER	3.72s
NPCL, |M| = 200	19.58s
NPCL, |M| = 500	31.25s
NPCL, |M| = 1000	47.99s
NPCL, |M| = 2000	84.86s
Table 8: Inference time with varying context sizes
Incompatibility with logits-based replay. The NPCL is incompatible with logits-based replay because of the stochasticity in the posterior induced by Monte Carlo sampling. Overcoming this could help boost the performance of the NPCL further over SOTA like DER [4] and  $\mathrm{DER}++$  [3].
In this paper, we propose Neural Processes for Continual Learning (NPCL), a hierarchical latent variable setup designed to jointly model the task-agnostic and task-specific data-generating functions in continual learning. We study the potential forgetting aspects in the NPCL and propose to regularize the previously learned distributions at a global and a per-task granularity. We demonstrate that using entropy as an uncertainty quantification metric helps the NPCL infer correct task heads and boost the performance of baseline experience replay to even surpass state-of-the-art deterministic models on several CL settings. Our robust ablations show the efficacy of the NPCL for model calibration measurement and few-shot replay in CL. We further study out-of-the-box applications of the uncertainty estimation capabilities of the NPCL for novel data identification and instance-level confidence evaluation. We conclude our ablations by listing the key limitations of the NPCL, which we hope could lay solid directions for further research on uncertainty-aware continual learning.
This work was partially supported by an ARC DECRA Fellowship DE230101591 awarded to Dong Gong. We acknowledge the reviewers for their valuable feedback.
[49] Jianfeng Wang, Thomas Lukasiewicz, Daniela Massiceti, Xiaolin Hu, Vladimir Pavlovic, and Alexandros Neophytou. Np-match: When neural processes meet semi-supervised learning. In International Conference on Machine Learning, pages 22919–22934. PMLR, 2022.
[50] Qi Wang and Herke Van Hoof. Doubly stochastic variational inference for neural processes with hierarchical latent variables. In International Conference on Machine Learning, pages 10018-10028. PMLR, 2020.
[51] Qi Wang and Herke van Hoof. Learning expressive meta-representations with mixture of expert neural processes. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=ju38DG3sbg6.
[52] Qingsen Yan, Dong Gong, Yuhang Liu, Anton van den Hengel, and Javen Qinfeng Shi. Learning bayesian sparse networks with full experience replay for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 109-118, 2022.
[53] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=Sk7KsfW0-.
[54] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pages 3987-3995. PMLR, 2017.
[55] Guodong Zhang, Shengyang Sun, David Duvenaud, and Roger Grosse. Noisy natural gradient as variational inference. In International conference on machine learning, pages 5852-5861. PMLR, 2018.
A.1 ELBO derivation for the NPCL
Borrowing the conventions from Sec. 3, for an incremental task  $0 \leq t \leq T$ , we assume the context  $\mathcal{C}$  and targets  $\mathcal{T}$  to comprise of samples from all  $t$  seen classes. Accordingly, we define these as  $\mathcal{C} = (X_{\mathcal{C}}^{0:t}, Y_{\mathcal{C}}^{0:t})$  and  $\mathcal{T} = (X_{\mathcal{T}}^{0:t}, Y_{\mathcal{T}}^{0:t})$ , respectively. To enforce the prior that both  $\mathcal{C}$  and  $\mathcal{T}$  follow the same distribution, we assume  $\mathcal{C}^t \subset \mathcal{T}^t$ , and therefore,  $\mathcal{C} \subset \mathcal{T}$ . In order to derive predictions  $Y_{\mathcal{T}}^{0:t}$  on  $X_{\mathcal{T}}^{0:t}$ , the NPCL relies on the context  $\mathcal{C}$  to build conditional priors  $p_{\theta}(z^G|\mathcal{C})$  and  $p_{\theta}(z^t|z^G,\mathcal{C}^t)$ , where  $p_{\theta}$  is the decoder. The decoder's objective thus boils down to maximizing the log-likelihood of the observations, i.e., the evidence  $\log p_{\theta}(Y_{\mathcal{T}}^{0:t}|X_{\mathcal{T}}^{0:t},\mathcal{C})$ . In the following, we derive the evidence lower bound (ELBO):
where the evidence is equal to the sum of the reconstruction likelihood  $\mathbb{E}_{q_{\phi}(z^G|\mathcal{T})}\big[\log p_{\theta}(Y_{\mathcal{T}}^{0:t}|X_{\mathcal{T}}^{0:t},\mathcal{C},z^{\mathcal{G}})\big]$  of the decoder and the KL divergence between the true posterior  $p_{\theta}(z^{G}|\mathcal{T})$  and the approximate posterior  $q_{\phi}(z^{G}|\mathcal{T})$  learned using the variational distribution, minus the prior matching term  $D_{\mathrm{KL}}\big(q_{\phi}(z^{G}|\mathcal{T})\| p_{\theta}(z^{G}|\mathcal{C})\big)$ . In particular, the NPCL learns two approximate distributions  $q_{\phi}(z^{G}|\mathcal{T})$  and  $q_{\phi}(z^{t}|z^{G},\mathcal{T}^{t})$ , that seek to estimate the global posterior  $p_{\theta}(z^{G}|\mathcal{T})$  and the task-specific posterior  $p_{\theta}(z^{t}|z^{G},\mathcal{T}^{t})$ . To realize the latter posterior, we introduce the hierarchy of task-specific latent variables  $z^{0:t}$ . This allows us to expand and derive a lower bound to the reconstruction likelihood as:
$$
\log p _ {\theta} (Y _ {\mathcal {T}} ^ {0: t} | X _ {\mathcal {T}} ^ {0: t}, \mathcal {C}, z ^ {G})
$$
(Reconstruction term)
(11a)
$$
= \mathbb {E} _ {\prod_ {0} ^ {t} q _ {\phi} (z ^ {t} | z ^ {G}, \mathcal {T} ^ {t})} \left[ \log \frac {p _ {\theta} (Y _ {\mathcal {T}} ^ {0 : t} , z ^ {0 : t} | X ^ {0 : t} , \mathcal {C} , z ^ {G})}{p _ {\theta} (z ^ {0 : t} | X _ {\mathcal {T}} ^ {0 : t} , Y _ {\mathcal {T}} ^ {0 : t} , \mathcal {C} , z ^ {G})} \right]
$$
(Introduce one-level latent hierarchy)
(11b)
$$
= \mathbb {E} _ {\Pi_ {0} ^ {t}} q _ {\phi} (z ^ {t} | z ^ {G}, \mathcal {T} ^ {t}) \left[ \log \int_ {0} ^ {t} \frac {p _ {\theta} (Y _ {\mathcal {T}} ^ {t} , z ^ {t} | X ^ {t} , \mathcal {C} ^ {t} , z ^ {G})}{p _ {\theta} (z ^ {t} | X _ {\mathcal {T}} ^ {t} , Y _ {\mathcal {T}} ^ {t} , \mathcal {C} ^ {t} , z ^ {G})} \right]
$$
(Integrate over individual tasks)
(11c)
$$
= \int_ {0} ^ {t} \mathbb {E} _ {q _ {\phi} (z ^ {t} | z ^ {G}, \mathcal {T} ^ {t})} \Big [ \log \frac {p _ {\theta} (Y _ {\mathcal {T}} ^ {t} | X _ {\mathcal {T}} ^ {t} , \mathcal {C} ^ {t} , z ^ {G} , z ^ {t}) p _ {\theta} (z ^ {t} | X _ {\mathcal {T}} ^ {t} , \mathcal {C} ^ {t} , z ^ {G})}{p _ {\theta} (z ^ {t} | \mathcal {T} ^ {t} , z ^ {G})} \Big ]
$$
(Chain rule of probability;  $\mathcal{C}\subset \mathcal{T}$
(11d)