IBCL: Zero-shot Model Generation under Stability-Plasticity Trade-offs

## Abstract

Abstract Algorithms that balance the stability-plasticity trade-off are well-studied in the continual learning literature. However, only a few of them focus on obtaining models for specified trade-off preferences. When solving the problem of continual learning under specific trade-offs (CLuST), state-of-the-art techniques leverage rehearsal-based learning, which requires retraining when a model corresponding to a new trade-off preference is requested. This is inefficient since there exist infinitely many different trade-offs, and a large number of models may be requested. As a response, we propose Imprecise Bayesian Continual Learning (IBCL), an algorithm that tackles CLuST efficiently. IBCL replaces retraining with constant-time convex combination. Given a new task, IBCL (1) updates the knowledge base in the form of a convex hull of model parameter distributions and (2) generates one Pareto-optimal model per given trade-off via convex combination without any additional training. That is, obtaining models corresponding to specified trade-offs via IBCL is zero-shot. Experiments whose baselines are current CLuST algorithms show that IBCL improves by at most 45% on average per task accuracy and by 43% on peak per task accuracy, while maintaining a near-zero to positive backward transfer. Moreover, its training overhead, measured by number of batch updates, remains constant at every task, regardless of the number of preferences requested. Details at: https://github.com/ibcl-anon/ibcl .

## 1 Introduction

Continual learning (CL), also known as lifelong machine learning, is a special case of multi-task learning, where tasks arrive in temporal sequence one-by-one [, , , ]. Two key properties matter for CL algorithms: stability and plasticity []. Here, stability means the ability to maintain performance on previous tasks, not forgetting what the model has learned, and plasticity means the ability to adapt to a new task. Unfortunately, these two properties are conflicting due to the multi-objective optimization nature of CL [, ]. For years, researchers have been balancing the stability-plasticity trade-off. However, few have discussed the problem of learning models for specifically given trade-off points. In this paper, we focus on such a problem, which we denote as CL under specific trade-offs (CLuST).

Why is CLuST important? In certain scenarios, it is important to explicitly specify how much stability and plasticity are needed. Let us consider an example of a movie recommendation system. The model is first trained to rate movies in the sci-fi genre. Then, a new genre, e.g. documentaries, is added by the movie company. The model needs to learn how to rate documentaries while not forgetting how to rate sci-fis. Training this model boils down to a CL problem.

The company now wants to build a recommendation system that adapts to users’ tastes in movies. For example, Alice has equal preferences over sci-fis and documentaries. Bob, however, wants to watch only documentaries and has no interest in sci-fis at all. Consequently, the company aims to train two customized models for Alice and Bob respectively, to predict how likely a sci-fi or a documentary is to be recommended. Based on individual preferences, Alice’s personal model should balance between the accuracy in rating sci-fis and rating documentaries, while Bob’s model allows to compromise the accuracy in rating sci-fis to achieve a high accuracy in rating documentaries. As new genres are added, users should be able to input their preferences over all available genres to obtain customized models.

Figure: Figure 1: A Bayesian view of a Pareto-optimal distribution $q^{\prime}$ and a non-Pareto-optimal distribution $q^{\prime\prime}$.
Refer to caption: extracted/5864649/figures_neurips/bayesian_view.png

To formalize the CLuST problem, we take a Bayesian perspective, where learnable model parameters are viewed as random variables [, , ]. As illustrated in Figure 1, we consider all parameter distributions living in a metric space. This metric can be any valid metric for distributions, such as the $2$-Wasserstein distance []. The figure shows an example of two tasks, with their ground-truth distributions being $q_{1}$ and $q_{2}$, respectively. From this setup, a distribution emphasizing stability (at task 2) is a distribution closer to $q_{1}$ than $q_{2}$, and one prioritizing plasticity is closer to $q_{2}$ than $q_{1}$. Notice that irrespective of the desired stability-plasticity trade-off, we want the distribution to be Pareto-optimal, which loosely means that there is no way to improve such a distribution by making it
closer to both $q_{1}$ and $q_{2}$. We can see that Pareto-optimality is equivalent to being inside the convex set enclosed by $q_{1}$ and $q_{2}$. For example, $q^{\prime}$ in the figure is a Pareto-optimal distribution, while $q^{\prime\prime}$ is not. With this setting, we can specify a trade-off point using a preference vector [, ] $\bar{w}=(w_{1},w_{2})$, where $w_{1},w_{2}\geq 0$ and $w_{1}+w_{2}=1$. The preferred Pareto-optimal distribution is therefore a convex combination $w_{1}q_{1}+w_{2}q_{2}$.

So far, researchers have already proposed the use of preference vectors to specify trade-off points in multi-task and continual learning [, , , ]. However, instead of using them as coefficients for convex combinations, state-of-the-art techniques use them as regularizers in rehearsal-based methods. That is, existing algorithms that aim to solve CLuST memorize some data $d_{i}$ for each task $i$ (for “rehearsal”), and let the loss at task $i$ be $l_{i}=\sum_{j=1}^{i}w_{j}l(d_{j})$, with $l$ being a generic loss function like cross-entropy. There are at least two drawbacks to this approach.
First, rehearsals have to retrain the entire model whenever we have a new trade-off preference. In plain words, these methods have a training overhead proportional to the number of preferences at each task. As there exist infinitely many possible preferences, this boils down to an efficiency issue when there is a large number of preferences, such as a large number of users in the movie recommendation example. It would be desirable if, instead of retraining, we could obtain the preferred models by constant-time, training-free operations. Moreover, rehearsals have to cache data, and stable performance on previous tasks depends on which data can be memorized.

Figure: Figure 2: The workflow of Imprecise Bayesian Continual Learning (IBCL). Here, we start from 1 prior, but in practice there may be more than 1 to reduce epistemic uncertainty.
Refer to caption: extracted/5864649/figures_neurips/ibcl_workflow.png

To overcome these shortcomings faced by CLuST algorithms, we propose
Imprecise Bayesian Continual Learning (IBCL), whose workflow is illustrated in Figure 2. At step 1, upon the arrival of a new task’s training data, IBCL update its knowledge base (that is, all information shared across tasks) in the form of a convex set of distributions with finitely many extreme elements (the elements that cannot be written as convex combinations of one another), called finitely generated credal set (FGCS) []. This is done by variational inference from the previous task’s learned distribution, and the learned distributions serve as extreme elements of the FGCS.
Each point in the FGCS corresponds to one Pareto-optimal distribution on the trade-off polytope of all tasks so far.
Then, at step 2, given any preference vector $\bar{w}$, IBCL selects the preferred distribution by convex combination. A parameter region is obtained as a highest density region (HDR) of the distribution selected, which is the smallest parameter set that contains the ground-truth model with high probability.

IBCL tackles the identified shortcomings as follows. First, IBCL replaces retraining in state-of-the-art with constant-time, zero-shot convex combination to generate models. It has constant training overhead per task (to update the FGCS), independent of the number of preferences. In addition, no cache of data is required, and therefore the stability of our model does not depend on the data memorized.
Experiments on image classification and NLP benchmarks support the effectiveness of IBCL. We find that IBCL improves on baselines by at most 45% in average per task accuracy and by 43% in peak per task accuracy, while maintaining a near-zero to positive backward transfer, with a constant training overhead regardless of number of preferences. We also show that IBCL has a sublinear memory growth along the number of tasks.

Overall, we have the following contributions: (1) We are the first to rigorously formulate the CLuST problem, which asks for efficiency upon a large number of preferences (Section 3). (2) We propose IBCL, a Bayesian CL algorithm to solve the CLuST problem (Section 4). (3) We experiment on standard image classification and NLP benchmarks to support our claims (Section 5).

## 2 Background

Our algorithm hinges upon the concepts of finitely generated credal set (FGCS) from Imprecise Probability (IP) theory
[, , ].

###### Definition 1 (Finitely Generated Credal Set) .

A convex set $\mathcal{Q}=\{q:q=\sum_{j=1}^{m}\beta^{j}q^{j}\text{, }\beta^{j}\geq 0\text{ }
\forall j\text{, }\sum_{j}\beta^{j}=1\}$ of probability distributions with finitely many extreme elements $\text{ex}[\mathcal{Q}]=\{q^{j}\}_{j=1}^{m}$ is called a finitely generated credal set (FGCS).

In other words, an FGCS is a convex hull of (finitely many) distributions.
We also borrow from the Bayesian literature the idea of highest density region (HDR) [].

###### Definition 2 (Highest Density Region) .

Let $\Theta$ be a set of interest, and $\alpha\in[0,1]$ be a significance level. Suppose that a (continuous) random variable $\theta\in\Theta$ has probability density function (pdf) $q$.
Then, the $(1-\alpha)$-HDR is the set $\Theta_{q}^{\alpha}$ such that $\int_{\Theta_{q}^{\alpha}}q(\theta)\text{d}\theta\geq 1-\alpha$ and $\int_{\Theta_{q}^{\alpha}}\text{d}\theta$ is a minimum.

Definition 2 tells us that if $\theta\sim q$, $\Pr_{\theta\sim q}[\theta\in\Theta_{q}^{\alpha}]\geq 1-\alpha$, and $\Theta_{q}^{\alpha}$ is the narrowest subset of $\Theta$ that guarantees this inequality. In other words, the HDR $\Theta_{q}^{\alpha}$ is the smallest subset of $\Theta$ where we can find the realization of random variable $\theta$ with high probability.
The concept of HDR is further explained in Appendix B with an illustration. Other related works, including multi-task and continual learning, are reviewed in Appendix C.

###### Definition 1 (Finitely Generated Credal Set) .

###### Definition 2 (Highest Density Region) .

## 3 Formulating the CLuST Problem

In this section we formalize the CLuST problem. We consider domain-incremental learning [] for classification models, with an unbounded number of stability-plasticity trade-off preferences at each task. The goal is to construct a learning algorithm with training overhead independent of the number of preferences, and that enjoys performance guarantees.

### 3.1 Assumptions

Let $\mathcal{X}$ be the space of inputs, and $\mathcal{Y}$ be the space of labels. Call $\Delta_{\mathcal{XY}}$ the space of all possible distributions on $\mathcal{X}\times\mathcal{Y}$. A task $i$ is associated with a distribution $p_{i}\in\Delta_{\mathcal{X}\mathcal{Y}}$, from which labeled data can be i.i.d. drawn.
We assume that all tasks are similar to each other.

###### Assumption 1 (Task Similarity) .

For all task $i$, $p_{i}\in\mathcal{F}$, where $\mathcal{F}$ is a convex subset of $\Delta_{\mathcal{XY}}$. Also, we assume that the diameter of $\mathcal{F}$ is some $r>0$, that is, $\sup_{p,q\in\mathcal{F}}\|p-q\|_{W_{2}}\leq r$,
where $\|\cdot\|_{W_{2}}$ denotes the $2$-Wasserstein distance.

The definition of and the reason for choosing the $2$-Wasserstein metric are given in Appendix D.
Assumption 1 is needed to mitigate the possible model misspecification, which in turn could lead to catastrophic forgetting even when Bayesian inference is carried out exactly ([] and Appendix E).
Under Assumption 1, for any two tasks $i$ and $j$, their underlying distributions $p_{i}$ and $p_{j}$ are “close enough”, i.e. $\|p_{i}-p_{j}\|_{W_{2}}\leq r$. Moreover, since $\mathcal{F}$ is convex, any convex combination of task distributions belongs to $\mathcal{F}$.
Next, we assume the parameterization of class $\mathcal{F}$.

###### Assumption 2 (Parameterization of Task Distributions) .

Every distribution $F$ in $\mathcal{F}$ is parameterized by $\theta$, a parameter belonging to a parameter space $\Theta$.

An example of a parameterized family that satisfies Assumption 1 is given in Appendix F. Notice that all tasks share the same input space $\mathcal{X}$ and label space $\mathcal{Y}$, so the learning is domain-incremental. We then formalize stability-plasticity trade-off preferences over tasks.

###### Definition 3 (Stability-plasticity Trade-off Preferences over Tasks) .

Consider $k$ tasks with underlying distributions $p_{1},p_{2},\dots,p_{k}$. We express a stability-plasticity trade-off preference (or simply, a preference) over them via a probability vector $\bar{w}=(w_{1},w_{2},\dots,w_{k})^{\top}$, that is, $w_{i}\geq 0$ for all $i\in\{1,\ldots,k\}$, and $\sum_{i=1}^{k}w_{i}=1$.

Based on this definition, given a preference $\bar{w}$ over all $k$ tasks encountered, the personalized model for the user aims to learn the distribution $p_{\bar{w}}:=\sum_{i=1}^{k}w_{i}p_{i}$.
It is the distribution associated with tasks $1,\ldots,k$ that also takes into account a preference over them. Since $p_{\bar{w}}$ is the convex combination of $p_{1},\dots,p_{k}$, thanks to Assumptions 1 and 2, we have that $p_{\bar{w}}\in\mathcal{F}$, and therefore it is also parameterized by some $\theta\in\Theta$. The learning procedure is the same as standard supervised domain-incremental learning. Given task $k$, we draw $n_{k}$ labeled examples i.i.d. from an unknown $p_{k}$. Then, we are given at least one user preference $\bar{w}$ over the $k$ tasks so far. The data drawn for task $k+1$ will not be available until we have finished learning models for all preferences at task $k$.

###### Assumption 1 (Task Similarity) .

###### Assumption 2 (Parameterization of Task Distributions) .

###### Definition 3 (Stability-plasticity Trade-off Preferences over Tasks) .

### 3.2 Main Problem

We aim to design a domain-incremental learning algorithm that generates one model per preference over tasks
, with unbounded number of preferences and tasks. Given a significance level $\alpha\in[0,1]$, at any task $k$, the algorithm should satisfy:

- 1.
Zero-shot preferred model generation. Let $\bar{w}$ be a preference over the $k$ tasks. When there are more than one preference $\bar{w}_{s}$, $s\in\{1,2,\ldots\}$, no training is needed for generating models, for all $s>1$. That is, the model generation for new preferences is zero-shot.
- 2.
Probabilistic Pareto-optimality. We want to identify the smallest subset of model parameters, $\Theta_{\hat{q}_{\bar{w}}}^{\alpha}\subset\Theta$ (written as $\Theta_{\bar{w}}^{\alpha}$ for notational convenience from now on), that the Pareto-optimal parameter $\theta^{\star}_{\bar{w}}$ (i.e. the ground-truth parameter of $p_{\bar{w}}$) belongs to with high probability, i.e.,
$\Pr_{\theta^{\star}_{\bar{w}}\sim\hat{q}_{\bar{w}}}[\theta^{\star}_{\bar{w}}
\in\Theta_{\bar{w}}^{\alpha}]\geq 1-\alpha$,
under a known $\hat{q}_{\bar{w}}$ over $\Theta$.
- 3.
Sublinear buffer growth. The memory overhead for the entire procedure should be growing sublinearly in the number of tasks.

## 4 Imprecise Bayesian Continual Learning

As shown in Figure 2, IBCL performs two steps at each task. First, it updates a knowledge base in form of an FGCS (Section 4.1). Second, it uses a convex combination of the extreme elements of the FGCS, instead of retraining, to zero-shot generate models under given preferences (Section 4.2).

### 4.1 FGCS Knowledge Base Update

As discussed in the Introduction, we take a Bayesian continual learning approach, that is, the parameter $\theta$ of distribution $p_{k}$ pertaining to task $k$
is viewed as a random variable distributed according to some distribution $q$. At the beginning of the analysis, we specify $m$ many such distributions, $\text{ex}[\mathcal{Q}_{0}]=\{q_{0}^{1},\ldots,q_{0}^{m}\}$. They are the ones that the designer deems plausible – a priori – for parameter $\theta$ of task $1$.
Upon observing data pertaining to task $1$, we learn a set $\mathcal{Q}^{tmp}_{1}$ of posterior parameter distributions and buffer them as extreme elements $\text{ex}[\mathcal{Q}_{1}]$ of the FGCS $\mathcal{Q}_{1}$ corresponding to task $1$. We proceed similarly for the successive tasks $i\geq 2$.

Figure: Algorithm 1 FGCS Knowledge Base Update

In Algorithm 1, at task $i$, we approximate $m$ posteriors $q_{i}^{1},\dots q_{i}^{m}$ via variational inference from buffered priors $q_{i-1}^{1},\dots q_{i-1}^{m}$ one-by-one
(line 3). However, we do not want to buffer all learned posteriors, so we use a distance threshold $d$ to exclude the posteriors that are similar to the distributions that are already buffered (lines 4 - 9). When a distribution similar to $q_{i}^{j}$ is found in the knowledge base, we remember to use it in place of $q_{i}^{j}$ in the future (line 8). The posteriors that are sufficiently different from the already buffered distributions are then
appended to the knowledge base (line 11).

Notice that Algorithm 1 ensures sublinear buffer growth in our problem formulation because at each task $i$ we only buffer $m_{i}$ new posterior models, with $0\leq m_{i}\leq m$. With a sufficiently large threshold $d$, the buffer growth can become constant after several tasks. The use of different threshold $d$’s is discussed in our ablation studies, see Section 5.2.

### 4.2 Zero-shot Generation of User Preferred Models

Next, after having updated the FGCS extreme elements for task $i$, we are given a set of user preferences. For each preference $\bar{w}$, we need to identify the Pareto-optimal parameter $\theta^{\star}_{\bar{w}}$ for the preferred data distribution $p_{\bar{w}}$. This procedure can be divided into two steps as follows.

First, we find the parameter distribution $\hat{q}_{\bar{w}}$ via a convex combination of the extreme elements in the knowledge base, whose weights correspond to the entries of preference vector $\bar{w}$. That is,

$$ $\begin{split}&\hat{q}_{\bar{w}}=\sum_{k=1}^{i}\sum_{j=1}^{m_{k}}\beta_{k}^{j}q _{k}^{j}\text{ where }\sum_{j=1}^{m_{k}}\beta_{k}^{j}=w_{k}\text{, and }\beta_ {k}^{j}\geq 0,\text{ for all }j\text{ and all }k.\end{split}$ (1) $$

Here, $q_{k}^{j}$ is a buffered extreme point of FGCS $\mathcal{Q}_{k}$, i.e. the $j$-th parameter posterior of task $k$. The weight $\beta_{k}^{j}$ of this extreme point is decided by preference vector entry $\bar{w}_{j}$. In implementation, if we have $m_{k}$ extreme elements stored for task $k$, we can choose equal weights $\beta_{k}^{1}=\dots=\beta_{k}^{m}=w_{k}/m_{k}$. For example, if we have preference $\bar{w}=(0.8,0.2)^{\top}$ on two tasks so far, and we have two extreme elements per task stored in the knowledge base, we can use $\beta_{1}^{1}=\beta_{1}^{2}=0.8/2=0.4$ and $\beta_{2}^{1}=\beta_{2}^{2}=0.2/2=0.1$.

As we can see from the following theorem, distribution $\hat{q}_{\bar{w}}$ is a parameter posterior corresponding to a preference elicitation via preference vector $\bar{w}$ over the tasks encountered so far.

###### Theorem 1 (Selection Equivalence) .

Selecting a precise distribution $\hat{q}_{\bar{w}}$ from $\mathcal{Q}_{i}$ is equivalent to specifying a preference weight vector $\bar{w}$ on $p_{1},\dots,p_{i}$.

Please refer to Appendix G for the proof. Theorem 1 entails that the selection of $\hat{q}_{\bar{w}}$ in Algorithm 2 is related to the correct parameterization of $p_{\bar{w}}\in\Delta_{\mathcal{XY}}$.

Second, we compute the HDR $\Theta_{\bar{w}}^{\alpha}\subset\Theta$ from $\hat{q}_{\bar{w}}$. This is implemented via a standard procedure that locates the smallest region in the parameter space whose enclosed probability mass is (at least) $1-\alpha$, according to $\hat{q}_{\bar{w}}$.
This procedure can be routinely implemented, e.g., in $\mathsf{R}$, using package $\mathsf{HDInterval}$ [].
As a result, we locate the smallest set of parameters $\Theta_{\bar{w}}^{\alpha}\subset\Theta$ associated with the preference $\bar{w}$. This subroutine is formalized in Algorithm 2, and one remark is that it does not require any training, i.e., we meet our goal of zero-shot preferred model generation of Section 3.2.

Figure: Algorithm 2 Preference HDR Computation

###### Theorem 1 (Selection Equivalence) .

### 4.3 Overall IBCL Algorithm and Analysis

From the two subroutines in Sections 4.1 and 4.2, we construct the overall IBCL algorithm as in Algorithm 3.

Figure: Algorithm 3 Imprecise Bayesian Continual Learning

For each task, in line 3, we use Algorithm 1 to update the knowledge base by learning $m$ posteriors from the current priors. Some of these posteriors will be cached and some will be substituted by a previous distribution in the knowledge base.
In lines 5-7, according to a user-given preference over all tasks so far, we obtain the HDR of the model associated with preference $\bar{w}$ in zero-shot via Algorithm 2. Notice that this HDR computation does not require the initial priors $\text{ex}[\mathcal{Q}_{0}]$, so we can discard them once the posteriors are learned in the first task. The following theorem ensures that IBCL locates the user-preferred Pareto-optimal model with high probability.

###### Theorem 2 (Probabilistic Pareto-optimality) .

Pick any $\alpha\in[0,1]$. The Pareto-optimal parameter $\theta^{\star}_{\bar{w}}$, i.e., the ground-truth parameter for $p_{\bar{w}}$, belongs to $\Theta_{\bar{w}}^{\alpha}$ with probability at least $1-\alpha$ under distribution $\hat{q}_{\bar{w}}$. In formulas, $\Pr_{\theta^{\star}_{\bar{w}}\sim\hat{q}_{\bar{w}}}\left[\theta^{\star}_{\bar{
w}}\in\Theta_{\bar{w}}^{\alpha}\right]\geq 1-\alpha$.

Theorem 2 gives us a $(1-\alpha)$-guarantee in obtaining Pareto-optimal models for given task trade-off preferences.
Consequently, the IBCL algorithm enjoys the probabilistic Pareto-optimality targeted by our main problem. Please refer to Appendix G for the proof.

###### Theorem 2 (Probabilistic Pareto-optimality) .

## 5 Experiments

### 5.1 Setup

Although there exist many baseline methods for CL, only a few baselines for CLuST exist. The following CLuST baselines are selected for comparison.

- 1.
Rehearsal-based. This is the state-of-the-art technique for CLuST []. These methods memorize a subset of training data of every task encountered so far. Task preferences are then given as weights to regularize the loss on each task’s memorized data.
We choose GEM [] and A-GEM [] as two typical rehearsal-based methods.
- 2.
Rehearsal-based, Bayesian. Since IBCL is a Bayesian method, we also compare it to a Bayesian technique, VCL []. We equip VCL with episodic memory to make it rehearsal-based, and so to be able to specify a preference. This approach has been used in [].
- 3.
Prompt-based. Prompt-based CL has never been used for CLuST and therefore not a state-of-the-art. Still, they are considered efficient modern CL techniques. Therefore, we made an attempt to specify preferences on L2P [], a prompt-based method, by training one learnable prompt prefix per task, and use a preference-weighted sum of the prompts at inference time.

We experiment on four standard continual learning benchmarks, including three image classification and one NLP: (i) 15 tasks in CelebA [] (with vs. without attributes), (ii) 10 tasks in Split CIFAR-100 [] (animals vs. non-animals), (iii) 10 tasks in TinyImageNet [] (animals vs. non-animals) and (iv) 5 tasks in 20NewsGroup [] (news related to computers vs. not related to computers). For the first three image benchmarks, features are first extracted by ResNet-18 []. For 20NewsGroup, features are extracted by TF-IDF [].

As in standard continual learning evaluation, after training on task $i$, we evaluate the accuracy on all testing data of previous tasks $j\in\{1,\dots,i\}$. To evaluate how well does a model address preferences, we randomly generate $n_{\text{prefs}}=10$ preferences per task, except for task 1, whose preference is always given by scalar $1$. Therefore, for each method, we obtain 10 models at each task, and we evaluate a preference-weighted sum of their accuracies on previous tasks. Finally, these preference-weighted accuracies are used to compute standard continual learning metrics: average per task accuracy, peak per task accuracy, and backward transfer []. Experiments are
run on Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz. Detailed setup can be found in Appendix H.1.

### 5.2 Results

Figure: Figure 3: Results of 20NewsGroup. Since VCL and IBCL produce probabilistic models, we sample 10 deterministic models for each. The solid blue curve illustrates the top performance of deterministic models by IBCL, and the dashed blue curve is the mean performance. The shaded blue region is the performance range by IBCL. The same illustration method is used for VCL in green color.
Refer to caption: extracted/5864649/figures_new/20news.png

Figure: Figure 4: Results of TinyImageNet. The illustration method is the same as in Figure 3.
Refer to caption: extracted/5864649/figures_new/tin.png

Our results support the claim that IBCL not only achieves high performance by probabilistic Pareto-optimality, but is also efficient with zero-shot generation of models.

Since VCL and IBCL output probabilistic models (BNNs and HDRs), we sample 10 deterministic models from each and compute the range of their performance metrics, illustrated as shaded areas in Figures 3 and 4. They represent performances on the 20NewsGroup and TinyImageNet, respectively. In these figures, we draw the curves of top performance and mean performance of the sampled deterministic models by VCL and IBCL as solid and dashed lines, respectively. Due to page limit, we show the results on Split CIFAR-100 and CelebA in Figure 9 and 10 in Appendix H.2.
From Figures 3, 4, 9 and 10, we can see that IBCL overall generates the model with top performance (high accuracy) in all cases, while maintaining little catastrophic forgetting (near-zero to positive backward transfer). This is due to the probabilistic Pareto-optimality guarantee. Statistically, IBCL improves on the baselines by at most 45% on average per task accuracy and by 43% on peak per task accuracy (compared to L2P in 20News). So far, to our knowledge, there is no discussion on how to specify a task trade-off preference in prompt-based continual learning, and we only make an attempt for L2P, which generally works poorly.

As illustrated in the figures, IBCL has a slightly negative backward transfer in the very beginning but then this value stays near-zero or positive. This shows that although IBCL may slightly forget the knowledge learned from the first task at the second task, it steadily retains knowledge afterwards. This may be due to the choice of the priors, of the likelihood, of the variational method to approximate the posterior, or to an intrinsic characteristic of our method. Given its relevance, we defer studying this phenomenon to future work. We can also see how, although VCL’s backward transfer is higher than IBCL’s in the first few tasks,
it eventually decreases and takes values that are nearly identical to, or smaller than, the IBCL ones. For 20NewsGroup, this happens after 5 tasks, for TinyImageNet after 3 taks, for Split CIFAR-100 after 10 tasks, and for CelebA after 2 tasks.

**Table 1: Training overhead comparison, measured as # of batch updates required at a task. Here, $n_{i}$: # of training data points at task $i$, $n_{\text{prefs}}$: # of preferences per task, $n_{\text{mem}}$: # of data points memorized per task in rehearsal, $n_{\text{priors}}$: # of priors in IBCL, $e$: # of epochs and $b$: batch size. Notice that the overhead of rehearsal based methods are proportional to $n_{\text{prefs}}$.**
|  |  | # batch updates at task $i$ | # batch updates at last task |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  |  | CelebA | CIFAR100 | TImgNet | 20News |  |
| Rehearsal | GEM | $n_{\text{prefs}}$<br>$\times(n_{i}+(i-1)\times n_{\text{mem}})$<br>$\times e/b$ | 99747 | 19532 | 13594 | 35313 |
| A-GEM |  |  |  |  |  |  |
| VCL |  |  |  |  |  |  |
| Prompt | L2P | $n_{i}\times e/b$ | 9538 | 1250 | 938 | 2907 |
| IBCL (ours) | $n_{\text{priors}}\times n_{i}\times e/b$ | 28614 | 3750 | 2814 | 8721 |  |

Table 1 shows the training overhead comparison measured in number of batch updates per task. We can see how IBCL’s overhead is independent of the number of preferences $n_{\text{prefs}}$ because it only requires training for the FGCS but not for the preferred models. Consequently, our experiments show that IBCL is able to maintain a constant training overhead per task, regardless of $n_{\text{prefs}}$ while achieving high performance. Although L2P also has this constant overhead, its performance is too poor to be acceptable.

Figure: Figure 5: Different $\alpha$’s on different preferences over the first two tasks in 20NewsGroup.
Refer to caption: extracted/5864649/figures/20news_alpha.png

Figure: Figure 6: Different $\alpha$’s on randomly generated preferences over all tasks in 20NewsGroup.
Refer to caption: extracted/5864649/figures/20news_alpha_2.png

Figure: Figure 7: Different $d$’s on 20NewsGroup and Split-CIFAR100. The buffer growth curves of $d=5e-3$ and $d=2e-3$ of 20NewsGroup are overlapping.
Refer to caption: extracted/5864649/figures/d_ablation.png

The main experiments are conducted with hyperparameters $\alpha=0.01$ and $d=0.002$. We also conduct two ablation studies. The first one is on different significance level $\alpha$ in Algorithm 2.

In Figure 5, we evaluate testing accuracy on three different $\alpha$’s over five different preferences (from $[0.1,0.9]$ to $[0.9,0.1]$) on the first two tasks of 20NewsGroup. For each preference, we uniformly sample 200 deterministic models from the HDR. We use the sampled model with the maximum L2 sum of the two accuracies to estimate the Pareto optimality under a preference. We can see that, as $\alpha$ approaches 0, we tend to sample closer to the Pareto front. This is because, with a smaller $\alpha$, HDRs become wider and we have a higher probability to sample Pareto-optimal models according to Theorem 2. For instance, when $\alpha=0.01$, we have a probability of at least $0.99$ that the Pareto-optimal solution is contained in the HDR.
Figure 6 shows that the performance drops as $\alpha$ increases, because we are more likely to sample poorly performing models from the HDR.
The second ablation study is on different thresholds $d$ in Algorithm 1. As $d$ increases, we are allowing more posteriors in the knowledge base to be reused. This will lead to memory efficiency at the cost of a performance drop. Figure 7 supports this trend.
With an appropriately selected $d$, we can guarantee that the model’s performance will not be overly affected, and that we save buffer memory.
For Split-CIFAR100, when $d=8e-3$, the buffer stops growing after task 6.

## 6 Conclusion

Overall, we propose a probabilistic continual learning algorithm, namely IBCL, to tackle the CLuST problem, where an unbounded number of stability-plasticity trade-off preferences may be requested at each task.
IBCL (i) guarantees that the Pareto-optimal model under a given preference can be sampled from the output HDR with high probability, (ii) zero-shot generates these preferred model, with training overhead not scaling up with number of preferences, and (iii) has a sublinear buffer growth in number of tasks.

Limitations of IBCL. Poorly performing models can also be sampled from IBCL’s HDRs. However, in practice, we can fine-tune $\alpha$ to shrink down the HDR to avoid poorly performing ones, as shown in the ablation studies.

Broader Impacts. IBCL is potentially useful in deriving user-customized models from large multi-task models, due to its not only guaranteed but also zero-shot preference addressing. These include large language models, recommendation systems and other applications.

## Appendix A Reason to adopt a Bayesian continual learning approach

Let $q_{0}(\theta)$ be our prior probability density/mass function (pdf/pmf) on parameter $\theta\in\Theta$ at time $t=0$. At time $t=1$, we collect data $(\bar{x}_{1},\bar{y}_{1})$ pertaining to task $1$, we elicit likelihood pdf/pmf $\ell_{1}(\bar{x}_{1},\bar{y}_{1}\mid\theta)$, and we compute $q_{1}(\theta\mid\bar{x}_{1},\bar{y}_{1})\propto q_{0}(\theta)\times\ell_{1}(
\bar{x}_{1},\bar{y}_{1}\mid\theta)$. At time $t=2$, we collect data $(\bar{x}_{2},\bar{y}_{2})$ pertaining to task $2$ and we elicit likelihood pdf/pmf $\ell_{2}(\bar{x}_{2},\bar{y}_{2}\mid\theta)$. Now we have two options.

- (i)
Bayesian Continual Learning (BCL): we let the prior pdf/pmf at time $t=2$ be the posterior pdf/pmf at time $t=1$. That is, our prior pdf/pmf is $q_{1}(\theta\mid\bar{x}_{1},\bar{y}_{1})$, and we compute $q_{2}(\theta\mid\bar{x}_{1},\bar{y}_{1},\bar{x}_{2},\bar{y}_{2})\propto q_{1}(
\theta\mid\bar{x}_{1},\bar{y}_{1})\times\ell_{2}(\bar{x}_{2},\bar{y}_{2}\mid
\theta)\propto q_{0}(\theta)\times\ell_{1}(\bar{x}_{1},\bar{y}_{1}\mid\theta)
\times\ell_{2}(\bar{x}_{2},\bar{y}_{2}\mid\theta)$;(^1^11Here we tacitly assume that the likelihoods are independent.)
- (ii)
Bayesian Isolated Learning (BIL): we let the prior pdf/pmf at time $t=2$ be a generic prior pdf/pmf $q_{0}^{\prime}(\theta)$. We compute $q_{2}^{\prime}(\theta\mid\bar{x}_{2},\bar{y}_{2})\propto q_{0}^{\prime}(\theta
)\times\ell_{2}(\bar{x}_{2},\bar{y}_{2}\mid\theta)$. We can even re-use the original prior, so that $q_{0}^{\prime}=q_{0}$.

As we can see, in option (i) we assume that the data generating process at time $t=2$ takes into account both tasks, while in option (ii) we posit that it only takes into account task $2$. Denote by $\sigma(X)$ the sigma-algebra generated by a generic random variable $X$. Let also $Q_{2}$ be the probability measure whose pdf/pmf is $q_{2}$, and $Q_{2}^{\prime}$ be the probability measure whose pdf/pmf is $q_{2}^{\prime}$. Then, we have the following.

###### Proposition 1 .

Posterior probability measure $Q_{2}$ can be written as a $\sigma(\bar{X}_{1},\bar{Y}_{1},\bar{X}_{2},\bar{Y}_{2})$-measurable random variable taking values in $[0,1]$, while posterior probability measure $Q_{2}^{\prime}$ can be written as a $\sigma(\bar{X}_{2},\bar{Y}_{2})$-measurable random variable taking values in $[0,1]$.

###### Proof.

Pick any $A\subset\Theta$. Then, $Q_{2}[A\mid\sigma(\bar{X}_{1},\bar{Y}_{1},\bar{X}_{2},\bar{Y}_{2})]=\mathbb{E}
_{Q_{2}}[\mathbbm{1}_{A}\mid\sigma(\bar{X}_{1},\bar{Y}_{1},\bar{X}_{2},\bar{Y}
_{2})]$, a $\sigma(\bar{X}_{1},\bar{Y}_{1},\bar{X}_{2},\bar{Y}_{2})$-measurable random variable taking values in $[0,1]$. Notice that $\mathbbm{1}_{A}$ denotes the indicator function for set $A$. Similarly, $Q_{2}^{\prime}[A\mid\sigma(\bar{X}_{2},\bar{Y}_{2})]=\mathbb{E}_{Q_{2}^{\prime
}}[\mathbbm{1}_{A}\mid\sigma(\bar{X}_{2},\bar{Y}_{2})]$, a $\sigma(\bar{X}_{2},\bar{Y}_{2})$-measurable random variable taking values in $[0,1]$. This is a well-known result in measure theory [].
∎

Of course Proposition 1 holds for all $t\geq 2$. Recall that the sigma-algebra $\sigma(X)$ generated by a generic random variable $X$ captures the idea of information encoded in observing $X$. An immediate corollary is the following.

###### Corollary 1 .

Let $t\geq 2$. Then, if we opt for BIL, we lose all the information encoded in $\{(\bar{X}_{i},\bar{Y}_{i})\}_{i=1}^{t-1}$.

In turn, if we opt for BIL, we obtain a posterior that is not measurable with respect to $\sigma(\{(\bar{X}_{i},\bar{Y}_{i})\}_{i=1}^{t})\setminus\sigma(\bar{X}_{t},
\bar{Y}_{t})$. If the true data generating process $p_{t}$ is a function of the previous data generating processes $p_{t^{\prime}}$, $t^{\prime}\leq t$, this leaves us with a worse approximation of the “true” posterior $Q^{\text{true}}\propto Q_{0}\times p_{t}$.

The phenomenon in Corollary 1 is commonly referred to as catastrophic forgetting. Continual learning literature is unanimous in labeling catastrophic forgetting as undesirable – see e.g. [, ]. For this reason, in this work we adopt a BCL approach. In practice, we cannot compute the posterior pdf/pmf exactly, and we will resort to variational inference to approximate them – an approach often referred to as Variational Continual Learning (VCL) []. As we shall see in Appendix E, Assumption 1 is needed in VCL to avoid catastrophic forgetting.

###### Proposition 1 .

###### Proof.

###### Corollary 1 .

### A.1 Relationship between IBCL and other BCL techniques

Like [, ], the weights in our Bayesian neural networks (BNNs) have Gaussian distribution with diagonal covariance matrix.
Because IBCL is rooted in Bayesian continual learning, we can initialize IBCL with a much smaller number of parameters to solve a complex task as long as it can solve a set of simpler tasks. In addition, IBCL does not need to evaluate the importance of parameters by measures such as computing the Fisher information, which are computationally expensive and intractable in large models.

#### A.1.1 Relationship between IBCL and MAML

In this section, we discuss the relationship between IBCL and the Model-Agnostic Meta-Learning (MAML) and Bayesian MAML (BMAML) procedures introduced in [, ], respectively. These are inherently different than IBCL, since the latter is a continual learning procedure, while MAML and BMAML are meta-learning algorithms. Nevertheless, given the popularity of these procedures, we feel that relating IBCL to them would be useful to draw some insights on IBCL itself.

In MAML and BMAML, a task $i$ is specified by a $n_{i}$-shot dataset $D_{i}$ that consists of a small number of training examples, e.g. observations $(x_{1_{i}},y_{1_{i}}),\ldots,(x_{n_{i}},y_{n_{i}})$. Tasks are sampled from a task distribution $\mathbb{T}$ such that the sampled tasks share the statistical regularity of the task distribution. In IBCL, Assumption 1 guarantees that the tasks $p_{i}$ share the statistical regularity of class $\mathcal{F}$. MAML and BMAML leverage this regularity to improve the learning efficiency of subsequent tasks.

At each meta-iteration $i$,

- 1.
Task-Sampling: For both MAML and BMAML, a mini-batch $T_{i}$ of tasks is sampled from the task distribution $\mathbb{T}$. Each task $\tau_{i}\in T_{i}$ provides task-train and task-validation data, $D^{\text{trn}}_{\tau_{i}}$ and $D^{\text{val}}_{\tau_{i}}$, respectively.
- 2.
Inner-Update: For MAML, the parameter of each task $\tau_{i}\in T_{i}$ is updated starting from the current generic initial parameter $\theta_{0}$, and then performing $n_{i}$ gradient descent steps on the task-train loss. For BMAML, the posterior $q(\theta_{\tau_{i}}\mid D^{\text{trn}}_{\tau_{i}},\theta_{0})$ is computed, for all $\tau_{i}\in T_{i}$.
- 3.
Outer-Update: For MAML, the generic initial parameter $\theta_{0}$ is updated by gradient descent. For BMAML, it is updated using the Chaser loss [, Equation (7)].

Notice how in our work $\bar{w}$ is a probability vector. This implies that if we fix a number of task $k$ and we let $\bar{w}$ be equal to $(w_{1},\ldots,w_{k})^{\top}$, then $\bar{w}\cdot\bar{p}$ can be seen as a sample from $\mathbb{T}$ such that $\mathbb{T}(p_{i})=w_{i}$, for all $i\in\{1,\ldots,k\}$.

Here lies the main difference between IBCL and BMAML.
In the latter the information provided by the tasks is used to obtain a refinement of the (parameter of the) distribution $\mathbb{T}$ on the tasks themselves. In IBCL, instead, we are interested in the optimal parameterization of the posterior distribution associated with $\bar{w}\cdot\bar{p}$. Notice also that at time $k+1$, in IBCL the support of $\mathbb{T}$ changes: it is $\{p_{1},\ldots,p_{k+1}\}$, while for MAML and BMAML it stays the same.

Also, MAML and BMAML can be seen as ensemble methods, since they use different values (MAML) or different distributions (BMAML) to perform the Outer-Update and come up with a single value (MAML) or a single distributions (BMAML). Instead, IBCL keeps distributions separate via FGCS, thus capturing the ambiguity faced by the designer during the analysis.

Furthermore, we want to point out how while for BMAML the tasks $\tau_{i}$ are all “candidates” for the true data generating process (dgp) $p_{i}$, in IBCL we approximate the pdf/pmf of $p_{i}$ with the product $\prod_{h=1}^{i}\ell_{h}$ of the likelihoods up to task $i$. The idea of different candidates for the true dgp is beneficial for IBCL as well: in the future, we plan to let go of Assumption 1 and let each $p_{i}$ belong to a credal set $\mathcal{P}_{i}$. This would capture the epistemic uncertainty faced by the agent on the true dgp.

To summarize, IBCL is a continual learning technique whose aim is to find the correct parameterization of the posterior associated with $\bar{w}\cdot\bar{p}$. Here, $\bar{w}$ expresses the developer’s preferences on the tasks. MAML and BMAML, instead, are meta-learning algorithms whose main concern is to refine the distribution $\mathbb{T}$ from which the tasks are sampled. While IBCL is able to capture the preferences of, and the ambiguity faced by, the designer, MAML and BMAML are unable to do so. On the contrary, these latter seem better suited to solve meta-learning problems. An interesting future research direction is to come up with imprecise BMAML, or IBMAML, where a credal set $\text{Conv}(\{\mathbb{T}_{1},\ldots,\mathbb{T}_{k}\})$ is used to capture the ambiguity faced by the developer in specifying the correct distribution on the possible tasks. The process of selecting one element from such credal set may lead to computational gains.

## Appendix B Highest Density Region

Equivalently to Definition 2, an HDR is defined as follows [].

###### Definition 4 .

Let $\Theta$ be a set of interest, and consider a significance level $\alpha\in[0,1]$. Suppose that a (continuous) random variable $\theta\in\Theta$ has probability density function (pdf) $q$.(^2^22Here too, for ease of notation, we do not distinguish between a random variable and its realization.)
The $\alpha$-level Highest Density Region (HDR) $\Theta^{\alpha}_{q}$ is the subset of $\Theta$ such that

$$ $\Theta^{\alpha}_{q}=\{\theta\in\Theta:q(\theta)\geq{q}^{\alpha}\},$ (2) $$

where
${q}^{\alpha}$ is a constant value. In particular, $q^{\alpha}$ is the largest constant such that $\Pr_{\theta\sim q}[\theta\in\Theta^{\alpha}_{q}]\geq 1-\alpha$.

Some scholars indicate HDRs as the Bayesian counterpart to the frequentist concept of confidence intervals. In dimension $1$, $\Theta^{\alpha}_{q}$ can be interpreted as the narrowest interval – or union of intervals – in which the value of the (true) parameter falls with probability of at least $1-\alpha$ according to distribution $q$. We give a simple visual example in Figure 8.

Figure: Figure 8: The $0.25$-HDR for a Normal Mixture density. This picture is a replica of Figure 1 in []. The geometric representation of “$75\%$ probability according to $q$” is the area between the pdf curve $q(\theta)$ and the horizontal bar corresponding to $q^{0.25}$. A higher probability coverage (according to $q$) would correspond to a lower constant, so $q^{\alpha}<q^{0.25}$, for all $\alpha<0.25$. In the limit, we recover $100\%$ coverage at $q^{0}=0$.
Refer to caption: x1.png

###### Definition 4 .

## Appendix C Additional Related Work

Multi-task Learning under Preferences. Learning for Pareto-optimal models under task performance trade-offs has been studied by researchers in multi-task learning [, ]. Various techniques have been applied to obtain models that address particular trade-off points [, , , ]. The idea of preferences on the trade-off points is introduced in multi-objective optimization [, ], and a preference can guide learning algorithms to search for a particular model. We borrow the formalization of preferences from [], where a preference is given by a vector of non-negative real weights $\bar{w}$, with each entry $w_{i}$ corresponding to task $i$.
That is, $w_{i}\geq w_{j}\iff i\succeq j$. This means that if $w_{i}\geq w_{j}$, then task $i$ is preferred to task $j$.
However, state-of-the-art algorithms require training one model per preference, imposing large overhead when there is a large number of preferences.

Continual Learning. Continual learning, also known as lifelong learning, is a special case of multi-task learning, where tasks arrive sequentially instead of simultaneously [, ].
In this paper, we leverage Bayesian inference in the knowledge base update [].
Like generic multi-task learning, continual learning also faces the stability-plasticity trade-off [],
which balances between performance on new tasks and resistance to catastrophic forgetting [].
Current methods identify models to address trade-off preferences by techniques such as loss regularization [], meaning at least one model needs to be trained per preference.

Researchers in CL have proposed various approaches to retain knowledge while updating a model on new tasks. These include modified loss landscapes for optimization [],
preservation of critical pathways via attention [],
memory-based methods [],
shared representations [],
and dynamic representations [].
Bayesian, or probabilistic methods such as variational inference are also adopted [, ].

## Appendix D 2-Wasserstein metric

In the main portion of the paper, we endowed $\Delta_{\mathcal{X}\mathcal{Y}}$ with the $2$-Wasserstein metric.
It is defined as

$$ $\begin{split}\|p-p^{\prime}\|_{W_{2}}\equiv W_{2}(p,p^{\prime}):=\sqrt{\inf_{ \gamma\in\Gamma(p,p^{\prime})}\mathbb{E}_{((x_{1},y_{1}),(x_{2},y_{2}))\sim \gamma}[d((x_{1},y_{1}),(x_{2},y_{2}))^{2}]},\end{split}$ (3) $$

where

- 1.
$p,p^{\prime}\in\Delta_{\mathcal{X}\mathcal{Y}}$;
- 2.
$\Gamma(p,p^{\prime})$ is the set of all couplings of $p$ and $p^{\prime}$. A coupling $\gamma$ is a joint probability measure on $(\mathcal{X}\times\mathcal{Y})\times(\mathcal{X}\times\mathcal{Y})$ whose marginals are $p$ and $p^{\prime}$ on the first and second factors, respectively;
- 3.
$d$ is the product metric endowed to $\mathcal{X}\times\mathcal{Y}$ [, Section 4.2].(^3^33We denote by $d_{\mathcal{X}}$ and $d_{\mathcal{Y}}$ the metrics endowed to $\mathcal{X}$ and $\mathcal{Y}$, respectively.)

We choose the $2$-Wasserstein distance for the ease of computation. In practice, when all distributions are modeled by Bayesian neural networks with independent Gaussian weights and biases, we have

$$ $\|q_{1}-q_{2}\|^{2}_{W_{2}}=\|\mu_{q_{1}}^{2}-\mu_{q_{2}}^{2}\|_{2}^{2}+\| \sigma_{q_{1}}^{2}\mathbf{1}-\sigma_{q_{2}}^{2}\mathbf{1}\|_{2}^{2},$ (4) $$

where $\|\cdot\|_{2}$ denotes the Euclidean norm, $\mathbf{1}$ is a vector of all $1$’s, and $\mu_{q}$ and $\sigma_{q}$ are respectively the mean and standard deviation of a multivariate normal distribution $q$ with independent dimensions, $q=\mathcal{N}(\mu_{q},\sigma^{2}_{q}I)$, $I$ being the identity matrix. Therefore, computing the $W_{2}$-distance between two distributions is equivalent to computing the difference between their means and variances.

## Appendix E Importance of Assumption 1

We need Assumption 1 in light of the results in []. There, the authors show that misspecified models can forget even when Bayesian inference is carried out exactly. By requiring that $\text{diam}(\mathcal{F})=r$, we control the amount of misspecification via $r$. In [], the authors design a new approach – called Prototypical Bayesian Continual Learning, or ProtoCL – that allows dropping Assumption 1 while retaining the Bayesian benefit of remembering previous tasks. Because the main goal of this paper is to come up with a procedure that allows the designer to express preferences over the tasks, we retain Assumption 1, and we work in the classical framework of Bayesian Continual Learning. In the future, we plan to generalize our results by operating with ProtoCL.(^4^44In [], the authors also show that if there is a task dataset imbalance, then the model can forget under certain assumptions. To avoid complications, in this work we tacitly assume that task datasets are balanced.)

## Appendix F An example of a parameterized family ℱ ℱ \mathcal{F} caligraphic_F

Let us give an example of a parameterized family $\mathcal{F}$. Suppose that we have one-dimensional data points and labels. At each task $i$, the marginal on $\mathcal{X}$ of $p_{i}$ is a Normal $\mathcal{N}(\mu,1)$, while the conditional distribution of label $y\in\mathcal{Y}$ given data point $x\in\mathcal{X}$ is a categorical $\text{Cat}(\vartheta)$. Hence, the parameter for $p_{i}$ is $\theta=(\mu,\vartheta)$, and it belongs to $\Theta=\mathbb{R}\times\mathbb{R}^{|\mathcal{Y}|}$. In this situation, an example of a family $\mathcal{F}$ satisfying Assumptions 1 and 2 is the convex hull of distributions that can be decomposed as we just described, and whose distance according to the $2$-Wasserstein metric does not exceed some $r>0$.

## Appendix G Proofs of the Theorems

###### Proof of Theorem 1 .

Without loss of generality, suppose we have encountered $i=2$ tasks so far, so the FGCS is $\mathcal{Q}_{2}$. Let $\text{ex}[\mathcal{Q}_{1}]=\{q_{1}^{j}\}_{j=1}^{m_{1}}$ and $\text{ex}[\mathcal{Q}_{2}]\setminus\text{ex}[\mathcal{Q}_{1}]=\{q_{2}^{j}\}_{j
=1}^{m_{2}}$.
Let $\hat{q}$ be any element of $\mathcal{Q}_{2}$. Then, there exists a probability vector $\bar{\beta}=(\beta_{1}^{1},\ldots,\beta_{1}^{m_{1}},\beta_{2}^{1},\ldots,\beta
_{2}^{m_{2}})^{\top}$ such that

$$ $\displaystyle\begin{split}\hat{q}&=\sum_{j=1}^{m_{1}}\beta_{1}^{j}q_{1}^{j}+ \sum_{j=1}^{m_{2}}\beta_{2}^{j}q_{2}^{j}\propto\hat{p}_{1}\sum_{j=1}^{m_{1}} \beta_{1}^{j}q_{0}^{j}+\hat{p}_{2}\sum_{j=1}^{m_{2}}\beta_{2}^{j}q^{j}_{0}. \end{split}$ (5) $$

Here, $\hat{p}_{i}=\prod_{k=1}^{i}\ell_{k}$, and $\ell_{k}$ is the likelihood at task $k$. It estimates the pdf of the true data generating process $p_{i}$ of task $i$. The proportional relationship in (5) is based on the Bayesian inference step (line 3, approximated via variational inference) of Algorithm 1.
We can then find a vector $\bar{w}=(w_{1}=\sum_{j=1}^{m_{1}}\beta_{1}^{j},w_{2}=\sum_{j=1}^{m_{2}}\beta_{
2}^{j})^{\top}$ that expresses the designer’s preferences over tasks $1$ and $2$. As we can see, then, the act of selecting a generic distribution $\hat{q}\in\mathcal{Q}_{2}$ is equivalent to specifying a preference vector $\bar{w}$ over tasks $1$ and $2$. This concludes the proof.
∎

###### Proof of Theorem 2 .

For maximum generality, assume $\Theta$ is uncountable.
Recall from Definition 2 that $\alpha$-level Highest Density Region $\Theta_{\bar{w}}^{\alpha}$ is defined as the subset of the parameter space $\Theta$ such that

$$ $\int_{\Theta_{\bar{w}}^{\alpha}}\hat{q}_{\bar{w}}(\theta)\text{d}\theta\geq 1- \alpha\quad\text{ and }\quad\int_{\Theta_{\bar{w}}^{\alpha}}\text{d}\theta \text{ is a minimum.}$ $$

We need $\int_{\Theta_{\bar{w}}^{\alpha}}\text{d}\theta$ to be a minimum because we want $\Theta_{\bar{w}}^{\alpha}$ to be the smallest possible region that gives us the desired probabilistic coverage. Equivalently, from Definition 4 we can write that $\Theta_{\bar{w}}^{\alpha}=\{\theta\in\Theta:\hat{q}_{\bar{w}}(\theta)\geq\hat{
q}_{\bar{w}}^{\alpha}\}$, where $\hat{q}_{\bar{w}}^{\alpha}$ is the largest constant such that $\Pr_{\theta\sim\hat{q}_{\bar{w}}}[\theta\in\Theta_{\bar{w}}^{\alpha}]\geq 1-\alpha$. Our result $\Pr_{\theta^{\star}_{\bar{w}}\sim\hat{q}_{\bar{w}}}[\theta^{\star}_{\bar{w}}
\in\Theta_{\bar{w}}^{\alpha})]\geq 1-\alpha$, then, comes from the fact that $\Pr_{\theta^{\star}_{\bar{w}}\sim\hat{q}_{\bar{w}}}[\theta^{\star}_{\bar{w}}
\in\Theta_{\bar{w}}^{\alpha})]=\int_{\Theta_{\bar{w}}^{\alpha}}\hat{q}_{\bar{w
}}(\theta)\text{d}\theta$, a consequence of a well-known equality in probability theory [].
∎

###### Proof of Theorem 1 .

###### Proof of Theorem 2 .

## Appendix H Details of Experiments

### H.1 Setup Details

We select 15 tasks from CelebA. All tasks are binary image classification on celebrity face images. Each task $i$ is to classify whether the face has an attribute such as wearing eyeglasses or having a mustache. The first 15 attributes (out of 40) in the attribute list [] are selected for our tasks. The training, validation and testing sets are already split upon download, with 162,770, 19,867 and 19,962 images, respectively. All images are annotated with binary labels of the 15 attributes in our tasks. We use the same training, validation and testing set for all tasks, with labels being the only difference.

We select 20 classes from CIFAR100 [] to construct 10 Split-CIFAR100 tasks []. Each task is a binary image classification between an animal class (label 0) and a non-animal class (label 1). The classes are (in order of tasks):

- 1.
Label 0: aquarium fish, beaver, dolphin, flatfish, otter, ray, seal, shark, trout, whale.
- 2.
Label 1: bicycle, bus, lawn mower, motorcycle, pickup truck, rocket, streetcar, tank, tractor, train.

That is, the first task is to classify between aquarium fish images and bicycle images, and so on. We want to show that the continual learning model incrementally gains knowledge of how to identify animals from non-animals throughout the task sequence. For each class, CIFAR100 has 500 training data points and 100 testing data points. We hold out 100 training data points for validation. Therefore, at each task we have 400 $\times$ 2 = 800 training data, 100 $\times$ 2 = 200 validation data and 100 $\times$ 2 = 200 testing data.

We also select 20 classes from TinyImageNet []. The setup is similar to Split-CIFAR100, with label 0 being animals and 1 being non-animals.

- 1.
Label 0: goldfish, European fire salamander, bullfrog, tailed frog, American alligator, boa constrictor, goose, koala, king penguin, albatross.
- 2.
Label 1: cliff, espresso, potpie, pizza, meatloaf, banana, orange, water tower, via duct, tractor.

The dataset already splits 500, 50 and 50 images for training, validation and testing per class. Therefore, each task has 1000, 100 and 100 images for training, validation and testing, respectively.

20NewsGroups [] contains news report texts on 20 topics. We select 10 topics for 5 binary text classification tasks. Each task is to distinguish whether the topic is computer-related (label 0) or not computer-related (label 1), as follows.

- 1.
Label 0: comp.graphics, comp.os.ms-windows.misc, comp.sys.ibm.pc.hardware, comp.sys.mac.hardware, comp.windows.x.
- 2.
Label 1: misc.forsale, rec.autos, rec.motorcycles, rec.sport.baseball, rec.sport.hockey.

Each class has different number of news reports. On average, a class has 565 reports for training and 376 for testing. We then hold out 100 reports from the 565 for validation. Therefore, each binary classification task has 930, 200 and 752 data points for training, validation and testing, on average respectively.

All data points are first preprocessed by a feature extractor. For images, the feature extractor is a pre-trained ResNet18 []. We input the images into the ResNet18 model and obtain its last hidden layer’s activations, which has a dimension of 512. For texts, the extractor is TF-IDF [] succeeded with PCA to reduce the dimension to 512 as well.

Each Bayesian network model is trained with evidence lower bound (ELBO) loss, with a fixed feed-forward architecture (input=512, hidden=64, output=1). The hidden layer is ReLU-activated and the output layer is sigmoid-activated. Therefore, our parameter space $\Theta$ is the set of all values that can be taken by this network’s weights and biases.

The three variational inference priors, learning rate, batch size and number of epcohs are tuned on validation sets. The tuning results are as follows. Here, “lr” stands for learning rate.

- 1.
CelebA: priors = $\{\mathcal{N}(0,0.2^{2}I)$, $\mathcal{N}(0,0.25^{2}I)$, $\mathcal{N}(0,0.3^{2}I)\}$, lr = $1e-3$, batch size = $64$, epochs = $10$.
- 2.
Split-CIFAR100: priors = $\{\mathcal{N}(0,2^{2}I)$, $\mathcal{N}(0,2.5^{2}I)$, $\mathcal{N}(0,3^{2}I)\}$, lr = $5e-4$, batch size = $32$, epochs = $50$.
- 3.
TinyImageNet: priors = $\{\mathcal{N}(0,2^{2}I)$, $\mathcal{N}(0,2.5^{2}I)$, $\mathcal{N}(0,3^{2}I)\}$, lr = $5e-4$, batch size = $32$, epochs = $30$.
- 4.
20NewsGroup: priors = $\{\mathcal{N}(0,2^{2}I)$, $\mathcal{N}(0,2.5^{2}I)$, $\mathcal{N}(0,3^{2}I)\}$, lr = $5e-4$, batch size = $32$, epochs = $100$.

For the baseline methods, we use exactly the same learning rate, batch sizes and epochs. For probabilistic baseline methods (VCL), we use the prior with the median standard deviation. For example, on CelebA tasks, VCL uses the normal prior $\mathcal{N}(0,0.25^{2}I)$.

For rehearsal-based baselines, the memory size per task for CelebA is 200, and for the rest is 50. Together with the numbers above, we can compute the numerical values in Table 1.

### H.2 Additional Results

In Figure 9 and 10, we provide visual representations of the performances of IBCL vs. the baseline methods on Split CIFAR-100 and CelebA, respectively. So far, to our knowledge, there is no discussion on how to specify a task trade-off preference in prompt-based continual learning, and we make an attempt by using a preference-weighted sum of all learned prompts in L2P. We can see how this approach generally works poorly, except for CelebA, where L2P performs nearly well as IBCL. We believe the performance by prompts trained in L2P depends on its frozen model, and how to use prompt-based methods to generate preference-specified models is still an open problem.

Figure: Figure 9: Results of Split CIFAR-100.
Refer to caption: extracted/5864649/figures_new/cifar100.png

Figure: Figure 10: Results of CelebA.
Refer to caption: extracted/5864649/figures_new/celeba.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 3
presentation: 1
questions: 1. What is the definition of $q^j$? Is it a real vector?
2. In definition 2, what is $\int$ is a minimum?
3. Are $\mathcal{X}, \mathcal{Y}$ subsets of the Euclidean space?
4. Variational inference can be computationally intensive? Will this algorithm become computationally infeasible in real-world settings?
soundness: 2
strengths: 1. The algorithm offers many favorable features, including the efficiency in model generation and sub-linear memory growth
2. It is innovative to investigate the problem through a Bayesian lens.
summary: The paper, introduces Imprecise Bayesian Continual Learning (IBCL) to address the Continual Learning under Specific Trade-offs (CLuST) problem. Traditional methods that balance stability and plasticity often rely on rehearsal-based learning, requiring retraining for each new trade-off, which is inefficient. IBCL offers a more efficient approach by constructing a convex hull of model parameter distributions and enabling zero-shot model generation for specific trade-offs without retraining. IBCL achieves this by updating a knowledge base in the form of a convex set of distributions and using convex combinations to generate Pareto-optimal models according to user-specified preferences. Experiments indicate that IBCL improves per-task accuracy and backward transfer compared to existing CLuST methods, with constant-time overhead and sub-linear memory growth.
weaknesses: I do not think this paper is well written enough for me to follow easily. First, some definitions are not formal. For example Definition 1, 2 should be written more formally. See questions 1 and 2 below. Second, some important concepts should be presented in detail with formulae. For example, in line 218, continual Bayesian learning appears without a formal introduction. Third, the words and phrases should be picked more carefully. For example, in line 8 of Algorithm 1, one should state: store xxx and use xxx when xxx instead of saying remember xxx and use xxx later on.

---

confidence: 2
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. What are the benefits of IBCL compared to linearly weight interpolation? For example, for a preference vector, the linear combination of model weights $\sum_{j=1}^m q_j \hat{\theta}_j$ itself induces a model (here the $\hat{\theta}_j$ can be any estimated model weights, for example, via empirical risk minimization). Some theoretical analysis and numerical experiments would be preferred.
soundness: 3
strengths: 1. The paper is written in a well-organized and self-contained way. The key concepts are clearly defined and sufficiently explained.
2. The idea of avoiding re-training models when receiving new preference vectors is smart. Transforming the convex combination of distributions into the convex combinations of posterior distributions of model parameters is natural, especially when the tasks are similar.
3. The numerical experiments verify the excellence of the algorithm. The code is also well-written.
summary: This paper models the problem of continual learning under specific trade-offs (CLuST) as a convex combination of previous tasks to the preference vector. The authors propose an algorithm named Imprecise Bayesian Continual Learning (IBCL) that transforms the convex combination of data distributions into the convex combination of model parameters under the framework of Bayesian learning. This algorithm is training-free for any newly arrived preference vectors compared to previous rehearsal-baed methods. The algorithm also performs well in numerical experiments.
weaknesses: 1. The contribution is restricted to the domain-incremental continual learning scenario.
2. The algorithm's effectiveness relies on a core assumption that there is a continuous mapping from the data distribution to the distribution of the ground-truth model parameters and that mapping is (approximately) linear. The authors should specify this reliance and perhaps give more discussions on the validity of the assumption (for example, the dependence on the model/prior choices and the dependence on the underlying data distributions).
3. The theory part does not have an in-depth algorithm analysis. Theorem 1 is about the modeling rather than the algorithm. Theorem 2 is unnecessary: a) the theorem relies on the assumption that the Pareto-optimal parameter follows the estimated posterior distribution, which directly assumes the correctness of the estimation; b) the theorem does not provide any useful information since the coverage guarantee is already defined in the high-density region (HDR). I suggest deleting Theorem 2 to avoid confusion. Some more in-depth discussions are preferred.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: Please refer to all the questions I mentioned in the weakness section. Overall, this paper gives an interesting idea but the proposed approach is not well explained and the results are not entire convincing. However, I am willing to raise my scores if my questions are well addressed.
soundness: 2
strengths: 1.	The idea of using Bayesian techniques, FGCS and the convex combination of posteriors of previous tasks to achieve zero-shot training seems interesting in the context of continual learning. 
2.	In the problems of continual learning under specific trade-offs (CluST), the proposed method significantly improves existing rehearsal-based and prompt-based algorithms.
summary: This paper studies the stability-plasticity trade-off for continual learning, with a focus on obtaining models for specified trade-off preferences. Differently from the typical rehearsal-based CL, which requires to retrain for every task and every preference, the authors propose a new paradigm named Imprecise Bayesian Continual Learning (UBCL) that replaces the retraining with weighted average of the extreme elements in the knowledge base, whose weights are provided by a preference vector. Since there is no additional training on the models, the entire procedure to achieve the provided trade-off is zero-shot. The main computation step is to compute the highest density region (HDR) under a distribution induced by weighted parameter posteriors of previous tasks. Empirically, the authors demonstrate that the proposed method outperforms existing algorithms in terms of accuracy significantly, with small overhead.
weaknesses: 1.	This paper is not well written. First, the motivation is not very clear. The paper considers CL under given preferences (i.e, weights). Although the paper gives some examples in recommendation systems, it does not talk much about how to get such weights. In addition, what is the setting of infinitely number of perferences? Some motivating examples are highly needed. 
2.	In terms of the presentation, there are also multiple places to be clarified. First, in line 208, there is a probability $\hat q_{\bar w}$ that is not defined at the first place. I find its definition later in equation (1). I suggest using a different notation or explain clearly here. In algorithm 1, some details or explanations may be needed, since some readers may be not familiar with this procedure. In theorem 2, how to find the pareto-optimal parameters $\theta^*_{\bar w}$? Is there any guarantee that the output of the proposed algorithm is parato-optimal? More clarifications should be provided.
3.	The design contains multiple heuristics and multiple hyparameters to be deciede. For example, in algorithm 1, lines 4-8 remove similar elements, based on a threshold d. How to select d in practice? Is the performance sensitive to it? Why such elimination is useful? In algorithm 2, why choosing $\beta^1_k=….=\beta^m_k$? Is it just because of simple implementation? How about other choices? Finally, the number $m$ of distributions for each task is selected to be 3 in the experiments. How about other choices? Is there any trade-off between the accuracy and efficiency?   
4.	The assumption made in this paper may be strong. In Assumption 1, it assumes all tasks have the same data distributions. It can happen that tasks share some level of similarity in data distribution, but they may involve different distribution components. In addition, in assumption 1, how to define $r$? This is because r can be either very large or very small. The final performance may be highly dependent on r. However, is there no such analysis. 
5.	Experiments are not entire convincing. Algorithms are compared in term of accuracy but how about their performance in forgetting? In addition, perhaps I missed something, but are there any ablation studies on the selection of d, m, \alpha and other hyperparameters?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: I’m curious about the “optimality” of the distribution to learn $p_{\bar{w}}$. Given a preference vector, the mixed probability $p_{\bar{w}}=\sum w_i p_i$ seems optimal under entropy loss; specifically, if $(X,Y)\sim p_{\bar{w}}$  then the probability $p_{\bar{w}}$ (conditional on $X$) minimizes the entropy loss when predicting $Y$ given $X$. However, if we consider an L2 loss, the optimal prediction becomes $\mathbb{E}[Y|X]$, a point estimate rather than a distribution. My questions are:

(1) Is the convex combination of true distributions $p_{\bar{w}}=\sum w_i p_i$ only optimal under entropy loss?

(2) If so, could the current method be extended to accommodate other loss functions, such as L2  or absolute loss?

Also, in the broader impact section, the potential of the proposed approach for large language models is mentioned. However, the experiments are conducted with a small neural network with only a single hidden layer. How does IBCL perform in terms of efficiency and effectiveness, particularly in training time per new task, when applied to larger models?
soundness: 3
strengths: The paper is well-written and easy to follow. The mathematical formulation of the problem is clear, and the figures effectively illustrate the proposed algorithm. Additionally, the paper provides analytical insights into the proposed algorithm, and the experimental results demonstrate its effectiveness.
summary: The paper introduces Imprecise Bayesian Continual Learning (IBCL), a zero-shot, Pareto-optimal model with sublinear buffer growth, designed to address the Continual Learning under Specific Trade-offs (CLuST) problem. It also provides a mathematical formulation of the CLuST problem and presents experiments to evaluate IBCL's effectiveness.
weaknesses: [1]I'm not sure about the applicability of the problem setup and the proposed model. It seems that the target distribution for learning (line 188) is only optimal under entropy loss for predictions. For more details, please refer to the questions section.

[2]A minor issue is the lack of ablation studies on the choices of prior distributions $q_0^j$'s and the parameter $\beta$'s. In the experiments, prior choices (lines 968-975) are fixed, and the $\beta$ values are uniformly chosen to recover the preference vector (Algorithm 2). Ablation studies on these choices would provide helpful insights.

Meta-review: The authors propose Imprecise Bayesian Continual Learning (IBCL) for zero-shot model generation under specified stability-plasticity trade-offs. Despite its innovative framing, there are several key weaknesses of this work. The writing is insufficiently rigorous, with unclear definitions and presentation gaps (e.g., vague assumptions and overly simplified algorithm descriptions). Critical theoretical guarantees (e.g., Pareto-optimality) are under-explored, and experimental validation lacks depth, especially regarding robustness to hyperparameter choices and task dissimilarities. While the idea has potential, these deficiencies limit its scientific contribution and practical applicability. Hence, I recommend rejection, as significant revisions and clarifications are needed before it reaches the standard for acceptance.

Decision: Reject