Derivation of ELBO for the static prior.
We seek to maximize the likelihood p(y^{1:T}) for all observed labels y^{1:T} . To derive the predictions, our framework uses the visual-aligned text features \tilde{\mathbf{t}}_{c}^{1:T} and the image inputs \mathbf{x} (see eq. ( 1 )). Our evidence is thus p(y^{1:T}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{1:T}) for which we derive the lower bound (ELBO). In the following, we denote the prior network as p_{\theta}(z^{t}) for which the true posterior is p_{\theta}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t}) . We approximate the true posterior using the variational posterior q_{\phi}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t}) . Our derivation ends up with the reconstruction term p_{\theta}(y^{t}|z^{t},\mathbf{x};\tilde{\mathbf{t}}_{c}^{t}) that can be seen as a deterministic function converting a given latent vector z^{t} and an input image \mathbf{x} into an observation y^{t} . For our CLIP-based variational framework, this deterministic function is the cosine similarity operation followed by the softmax application (Eq. ( 3b )).

\log\; p_{\theta}(y^{1:T}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{1:T}) (Log-likelihood of evidence) =\log p_{\theta}(y^{1:T}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{1:T})\int q_{\phi}(z^{1:T}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{1:T})dz^{1:T} \big{(}\because\int q_{\phi}(z^{1:T}|x^{1:T})dz^{1:T}=1\big{)} =\int q_{\phi}(z^{1:T}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{1:T})\big{(}\log p_{\theta}(y^{1:T}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{1:T})\big{)}dz^{1:T} (Bring evidence into integral) =\mathbb{E}_{q_{\phi}(z^{1:T}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{1:T})}\big{[}\log p_{\theta}(y^{1:T}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{1:T})\big{]} (Definition of Expectation) =\sum_{t=1}^{T}\bigg{[}\mathbb{E}_{q_{\phi}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}\big{[}\log p_{\theta}(y^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})\big{]}\bigg{]} (Rewrite using sum) =\sum_{t=1}^{T}\bigg{[}\mathbb{E}_{q_{\phi}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}\Big{[}\log\frac{p_{\theta}(y^{t},z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}{p_{\theta}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}\Big{]}\bigg{]} (Re-introduce $z^{t$ by Chain rule of probability}) =\sum_{t=1}^{T}\bigg{[}\mathbb{E}_{q_{\phi}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}\Big{[}\log\frac{p_{\theta}(y^{t},z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})q_{\phi}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}{p_{\theta}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})q_{\phi}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}\Big{]}\bigg{]} ( Multiply by 1 =\frac{q_{\phi}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}{q_{\phi}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}) =\sum_{t=1}^{T}\bigg{[}\mathbb{E}_{q_{\phi}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}\Big{[}\log\frac{p_{\theta}(y^{t},z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}{q_{\phi}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}\Big{]}+\mathbb{E}_{q_{\phi}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}\Big{[}\log\frac{q_{\phi}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}{p_{\theta}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}\Big{]}\bigg{]} (Split the expectation) =\sum_{t=1}^{T}\bigg{[}\mathbb{E}_{q_{\phi}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}\Big{[}\log\frac{p_{\theta}(y^{t},z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}{q_{\phi}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}\Big{]}+\mathbb{D}_{KL}\big{(}q_{\phi}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})\|p_{\theta}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})\big{)} (Definition of KL divergence) \geq\sum_{t=1}^{T}\bigg{[}\mathbb{E}_{q_{\phi}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}\Big{[}\log\frac{p_{\theta}(y^{t},z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}{q_{\phi}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}\Big{]}\bigg{]} (\because KL divergence\geq 0) \geq\sum_{t=1}^{T}\bigg{[}\mathbb{E}_{q_{\phi}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}\Big{[}\log\frac{p_{\theta}(y^{t}|z^{t},\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})p_{\chi}(z^{t})}{q_{\phi}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}\Big{]}\bigg{]} (Chain rule of probability) \geq\sum_{t=1}^{T}\bigg{[}\mathbb{E}_{q_{\phi}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}\Big{[}\log p_{\theta}(y^{t}|z^{t},\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})\Big{]}+\mathbb{E}_{q_{\phi}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}\Big{[}\frac{p_{\chi}(z^{t})}{q_{\phi}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}\Big{]}\bigg{]} (Split the Expectation) \geq\sum_{t=1}^{T}\bigg{[}\mathbb{E}_{q_{\phi}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}\Big{[}\log p_{\theta}(y^{t}|z^{t},\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})\Big{]}-\mathbb{D}_{KL}\big{(}q_{\phi}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})\|p_{\chi}(z^{t})\big{)}\bigg{]} (Definition of KL divergence)