Prediction-aware variational modelling
We are interested in modeling the stochastic processes that generate the labels y for the inputs \mathbf{x} of a CL task t . To this end, we assume a prior distribution p_{\chi} over the text features \{p_{\chi}(\mathbf{t}_{c})\}_{c=1}^{C} . In the feature space, we use the reparameterization trick [ 32 ] to represent \mathbf{t}_{c}(\mathbf{p})\sim p_{\chi}(\mathbf{t}_{c}(\mathbf{p})) as a linear combination of the deterministic text features g(\mathbf{t}_{c}) and a stochastic latent variable z\sim p_{\chi} , which can be readily plugged into Eq. ( 1 ):

\mathbf{t}_{c}(\mathbf{p}) =g(\mathbf{t}_{c}(\mathbf{p}))+z\mid z\sim p_{\chi}, (3a) p(y_{i}|\mathbf{x}) =\int_{\chi}\frac{\exp\big{(}\langle f(\mathbf{x})^{T},\mathbf{t}_{c}(\mathbf{p})\rangle\big{)}}{\sum_{c=1}^{|C^{t}|}\exp\big{(}\langle f(\mathbf{x})^{T},\mathbf{t}_{c}(\mathbf{p})\rangle\big{)}}p(p_{\chi}(\mathbf{t}_{c}(\mathbf{p})))d\chi, (3b)

where g(\cdot) is defined in Eq. ( 2 ). By modeling the generative processes behind the inputs,
Eq. ( 3b ) injects data-driven uncertainties into the predictions. It also offers us further advantages over other variational frameworks, such as [ 12 ] .
First, as the latent variable z is used to infer the logits, it favors generalization by encoding prediction-aware information. Second, Eq. ( 3b ) is prompt-agnostic as the text features could be derived from any existing soft [ 93 , 74 , 29 ] or hard [ 70 , 17 ] prompt-based method. Third, defining the distribution over the encoder outputs rather than the prompts helps inherit the computational efficiency of adapters.

To deal with the intractability of the marginal likelihood in Eq. ( 3b ), one could optimize for an evidence lower bound (ELBO) using a variational posterior q_{\phi} that matches the prior p_{\chi} based on the KL-divergence loss \mathbb{D}_{\text{KL}} :

\log p(y|\mathbf{x})\geq\mathbb{E}_{q_{\phi}(z|\mathbf{t}_{c})}[\log p(y|\mathbf{x},z)]-\mathbb{D}_{KL}\big{(}q_{\phi}(z|\mathbf{t}_{c})\|p_{\chi}\big{)}

Following standard practices [ 32 , 18 ] , q_{\phi} can be modeled as a Gaussian distribution \mathcal{N}(\mu(\mathbf{t}_{c}),\sigma(\mathbf{t}_{c})) whose mean \mu and standard deviation \sigma are parameterized by linear adapter layers – we thus refer to [\mu;\sigma] as probabilistic adapter. A simple choice for the static prior p_{\chi} is \mathcal{N}(\mathbf{0},\mathbf{I}) . Lastly, a finite set of Monte-Carlo (MC) samples \{z_{m}\}_{m=1}^{M} can be drawn from the posterior by reparameterization [ 32 ] .

However, finetuning adapters for CL is not straightforward. Namely, we have the overhead of search for the dataset-specific residual ratio \alpha (see Eq. ( 2 )) [ 17 , 87 ] . This has particularly worse implications for a probabilistic adapter like ours, where a too large \alpha can inject enough noise to corrupt the pre-trained representations to the point of catastrophic forgetting (see Fig. 2(a) ). To preserve our property \# B (Sec. 1 ) for backward compatibility with existing prompting techniques, we first seek to have no additional overhead of hyperparameter search for the residual ratio. Subsequently, we use \alpha=\beta=1 through our work.