Derivation of ELBO for the data-driven prior.
Similar to App. 10 , we start with the log-likelihood of the evidence which now involves conditioning on an additional context set \mathcal{C}^{1:T} . The t- th task context set is used to condition our prior network p_{\theta}(z^{t}|\mathcal{C}^{t}) . Following the standard practices of other data-driven prior frameworks [ 18 , 27 ] , we introduce parameter-sharing between our conditional prior and variational posterior networks. This allows us to replace our prior network with the variational posterior network q_{\phi}(z^{t}|\mathcal{T}^{t}) , where \mathcal{T} is the target set for task t .

\log \;p_{\theta}(Y_{\mathcal{T}}^{1:T}|\mathbf{x}_{\mathcal{T}}^{1:T},\mathcal{C}^{1:T}) (Log-likelihood of evidence) =\log p_{\theta}(Y_{\mathcal{T}}^{1:T}|\mathbf{x}_{\mathcal{T}}^{1:T},\mathcal{C}^{1:T})\int q_{\phi}(z^{1:T}|\mathbf{x}_{\mathcal{T}}^{1:T},\mathcal{C}^{1:T})dz^{1:T} \big{(}\because\int q_{\phi}(z^{1:T}|\mathbf{x}_{\mathcal{T}}^{1:T},\mathcal{C}^{1:T})dz^{1:T}=1\big{)} =\int q_{\phi}(z^{1:T}|\mathbf{x}_{\mathcal{T}}^{1:T},\mathcal{C}^{1:T})\big{(}\log p_{\theta}(Y_{\mathcal{T}}^{1:T}|\mathbf{x}_{\mathcal{T}}^{1:T},\mathcal{C}^{1:T})\big{)}dz^{1:T} (Bring evidence into integral) =\mathbb{E}_{q_{\phi}(z^{1:T}|\mathcal{T}^{1:T})}[\log p_{\theta}(Y_{\mathcal{T}}^{1:T}|\mathbf{x}_{\mathcal{T}}^{1:T},\mathcal{C}^{1:T})] (By definition) =\sum_{t=1}^{T}\bigg{[}\mathbb{E}_{q_{\phi}(z^{t}|\mathcal{T}^{t})}[\log p_{\theta}(Y_{\mathcal{T}}^{t}|\mathbf{x}_{\mathcal{T}}^{t},\mathcal{C}^{t})]\bigg{]} (Rewrite using sum) =\sum_{t=1}^{T}\bigg{[}\mathbb{E}_{q_{\phi}(z^{t}|\mathcal{T}^{t})}\Big{[}\log\frac{p_{\theta}(Y_{\mathcal{T}}^{t},z^{t}|\mathbf{x}_{\mathcal{T}}^{t},\mathcal{C}^{t})}{p_{\theta}(z^{t}|\mathbf{x}_{\mathcal{T}}^{t},Y_{\mathcal{T}}^{t},\mathcal{C}^{t})}\Big{]}\bigg{]} (Re-introduce\;z^{t} by Chain rule of probability) =\sum_{t=1}^{T}\bigg{[}\mathbb{E}_{q_{\phi}(z^{t}|\mathcal{T}^{t})}\Big{[}\log\frac{p_{\theta}(Y_{\mathcal{T}}^{t}|z^{t},\mathbf{x}_{\mathcal{T}}^{t},\mathcal{C}^{t})\;p_{\theta}(z^{t}|\mathcal{C}^{t})}{p_{\theta}(z^{t}|\mathcal{T}^{t})}\Big{]}\bigg{]} (By Chain rule of probability;\;\mathcal{C}\subset\mathcal{T}) =\sum_{t=1}^{T}\bigg{[}\mathbb{E}_{q_{\phi}(z^{t}|\mathcal{T}^{t})}\Big{[}\log\frac{p_{\theta}(Y_{\mathcal{T}}^{t}|z^{t},\mathbf{x}_{\mathcal{T}}^{t},\mathcal{C}^{t})\;p_{\theta}(z^{t}|\mathcal{C}^{t})\;q_{\phi}(z^{t}|\mathcal{T}^{t})}{p_{\theta}(z^{t}|\mathcal{T}^{t})\;q_{\phi}(z^{t}|\mathcal{T}^{t})}\Big{]}\bigg{]} (Equivalent fraction) =\sum_{t=1}^{T}\bigg{[}\mathbb{E}_{q_{\phi}(z^{t}|\mathcal{T}^{t})}\Big{[}\log p_{\theta}(Y_{\mathcal{T}}^{t}|z^{t},\mathbf{x}_{\mathcal{T}}^{t},\mathcal{C}^{t})\Big{]} \phantom{={}}\phantom{={}}\phantom{={}}+\mathbb{E}_{q_{\phi}(z^{t}|\mathcal{T}^{t})}\Big{[}\log\frac{p_{\theta}(z^{t}|\mathcal{C}^{t})}{q_{\phi}(z^{t}|\mathcal{T}^{t})}\Big{]}+\mathbb{E}_{q_{\phi}(z^{t}|\mathcal{T}^{t})}\Big{[}\log\frac{q_{\phi}(z^{t}|\mathcal{T}^{t})}{p_{\theta}(z^{t}|\mathcal{T}^{t})}\Big{]}\bigg{]} (Split the expectation) =\sum_{t=1}^{T}\bigg{[}\mathbb{E}_{q_{\phi}(z^{t}|\mathcal{T}^{t})}\Big{[}\log p_{\theta}(Y_{\mathcal{T}}^{t}|z^{t},\mathbf{x}_{\mathcal{T}}^{t},\mathcal{C}^{t})\Big{]} \phantom{={}}\phantom{={}}\phantom{={}}-\mathbb{D}_{KL}\big{(}q_{\phi}(z^{t}|\mathcal{T}^{t})\|p_{\theta}(z^{t}|\mathcal{C}^{t})\big{)}+\mathbb{D}_{KL}\big{(}q_{\phi}(z^{t}|\mathcal{T}^{t})\|p_{\theta}(z^{t}|\mathcal{T}^{t})\big{)}\bigg{]} (By definition of KL divergence) \geq\sum_{t=1}^{T}\bigg{[}\mathbb{E}_{q_{\phi}(z^{t}|\mathcal{T}^{t})}\Big{[}\log p_{\theta}(Y_{\mathcal{T}}^{t}|z^{t},\mathbf{x}_{\mathcal{T}}^{t},\mathcal{C}^{t})\Big{]}-\mathbb{D}_{KL}\big{(}q_{\phi}(z^{t}|\mathcal{T}^{t})\|p_{\theta}(z^{t}|\mathcal{C}^{t})\big{)}\bigg{]} (\becauseKL divergence\geq 0)