Generating Prompts in Latent Space for Rehearsal-free Continual Learning

Authors: chengyi  yang
wentao  liu
shisong  chen
jiayin  qi
aimin 2024 zhou
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-04 18:56:35

Abstract: Continual learning emerges as a framework that trains the model on a sequence of tasks without forgetting previously learned knowledge, which has been applied in multiple multimodal scenarios. Recently, prompt-based continual learning has achieved excellent domain adaptability and knowledge transfer through prompt generation. However, existing methods mainly focus on designing the architecture of a generator, neglecting the importance of providing effective guidance for training the generator. To address this issue, we propose Generating Prompts in Latent Space (GPLS), which considers prompts as latent variables to account for the uncertainty of prompt generation and aligns with the fact that prompts are inserted into the hidden layer outputs and exert an implicit influence on classification. GPLS adopts a trainable encoder to encode task and feature information into prompts with reparameterization technique, and provides refined and targeted guidance for the training process through the evidence lower bound (ELBO) related to Mahalanobis distance. Extensive experiments demonstrate that GPLS achieves state-of-the-art performance on various benchmarks. Our code is available at https://github.com/Hifipsysta/GPLS.

Main paper: 
 introduction: 
 Continual learning (CL) aims to mitigate the phenomenon of forgetting old knowledge that occurs when a neural network learns multiple sequentially arriving tasks, which is also known as catastrophic forgetting [11, 29] . Over the past year, continual learning has been applied in multiple multimodal tasks, including crossmodal retrieval [56, 58] , visual question answering [23, 37, 59 ] and visual-language model [60, 61] .In the early stage, researches on CL primarily concentrated on different training strategies, including designing regularization terms for objective function [1, 20, 24, 32, 42] , learning representative samples from old tasks repetitively [16, 22, 36, 38, 43, 50] , increasing parameters when new tasks arrive [15, 25, 48, 57] , and projecting gradients onto the orthogonal direction of old features [5, 10, 26, 27, 39, 40, 46] . The emergence of pre-trained Vision Transformer (ViT) [8] has led to the introduction of prompt-based continual learning [50] , which has demonstrated superior performance compared to the above methodologies.In CL scenarios, the utilization of prompt-based methods provides the advantage of maintaining the frozen state of the backbone parameters throughout the training process. As a result, the backbone parameters remain unchanged from their optimal state for the previous task, thereby mitigating the risk of forgetting, which is a primary concern when learning new tasks. Nonetheless, continual prompt learning encounters challenges related to its dependence on Figure 1 : Illustration of the localization, concentration and indirectness in prompt generation. The number of prompts corresponds to the quantity of transformer blocks, with each prompt being inserted into the input of a distinct transformer layer. While prompt generator is unique and it generates prompts for multiple transformer blocks. Therefore, the trainable parameters are concentrated in a certain area. a prompt pool. Specifically, utilizing prompt pool requires setting hyperparameters that are difficult to determine (e.g. pool size). More importantly, the number of prompts is commonly much smaller than the number of training instances, which limits the ability of prompts to offer fine-grained guidance to the backbone network.To avoid dependence on prompt pools, two different solutions have been proposed almost simultaneously: prompt decomposition [41] and prompt generation [17] . Prompt generation demonstrates better performance in continual learning due to its enhanced domain adaptability. However, we observed that existing prompt generation method lacked direct guidance or constraints on the process of prompt generation. For instance, Domain-Adaptive Prompt (DAP) [17] optimizes the prompt generator just by backpropagating the gradient of final cross-entropy loss, without directly supervising the process of prompt generation. The most direct evidence is that no variables directly related to the prompt generator were considered in the objective function.However, training a prompt generator only through cross-entropy loss is even more challenging compared to training prompts directly, which is determined by the three major properties of prompt generation methods (also see Figure 1 ). ( 1) Localization: Cross-entropy serves as a global loss metric, while prompt learning needs to focus on the impact of local parameter variations on classification results, because the backbone parameters are frozen and the trainable generator parameters are concentrated in one or a few narrow regions.(2) Concentration: In prompt generation, the concentration of trainable parameters is higher compared to prompt optimization approaches. Specifically, the trainable parameters in prompt optimization methods are distributed across the inputs of various transformer blocks, while the learnable parameters for prompt generation are contained in the unique prompt generator, irrespective of the quantity of transformer blocks utilized. (3) Indirectness: Parameter updating in prompt optimization methods can directly affect the classification results, while parameter updating in the prompt generator only exert an indirect influence on classification results via generated prompts.Regrettably, several studies [2, 13, 30, 54] have shown that parameter sparsity is beneficial for continual learning. In light of these characteristics, we propose Generating Prompts in Latent Space (GPLS) for continual learning. GPLS utilizes an encoder to encode task and feature information into prompts, and provides enhanced and targeted guidance for the training process through variational inference. We conceptualize the generated prompts as observations follow a distribution, which can better describe their uncertainty. Notably, prompts are considered as latent variables as they are inserted into the inputs of different transformer blocks, which is also the hidden layer outputs. Therefore, the problem of prompt generation is reframed as the task of estimating the distribution in latent space. In summary, our main contributions are as follows:(1) We consider prompts as latent variables following a distribution, which enhances the ability to capture the uncertainty of generated prompts and also conforms to the fact that prompts are inserted into the hidden layer outputs.(2) We propose a novel method called Generating Prompts in Latent Space (GPLS) for continual learning, which transforms the issue of prompt generation into the distribution estimation in latent space and generates prompts via a trainable encoder with reparameterization technique.(3) We derive the evidence lower bound (ELBO) for prompt generation and design an objective function related to Mahalanobis distance that offers refined guidance for the encoder to generate better prompts.(4) Extensive experiments show that GPLS achieves state-of-theart performance on five standard CL benchmarks, especially on DomainNet with a large number of samples and categories. 
related work 2.1 continual learning: 
 Continual learning aims to mitigate the forgetting of knowledge about old tasks after learning new tasks, which can be categorized into several categories according to the principles of algorithm design. Regularization-based approaches [1, 20, 24, 32, 42] mitigate catastrophic forgetting by restricting the parameter updating when the model learns a new task so that it cannot deviate from the optimal parameters on previous tasks. However, these methods also limit the learning capacity of models in the process of constraining parameters. Expansion-based methods [15, 25, 48, 57] maintain optimal parameters for old tasks and enhance the plasticity of the model by learning new knowledge through the newly added network parameters. Replay-based methods [16, 22, 36, 38, 43, 50] store a portion of previously trained data in a memory buffer and combine it with new training data when learning the new task. Although these two type of approaches achieve better performances, they come with higher training costs and increased graphics memory as the number of tasks grows. Projection-based methods [5, 10, 26, 27, 39, 40, 46] project the gradients of current task onto the orthogonal direction of old task features to prevent gradient interference between tasks. However, these methods perform singular value decomposition (SVD) on representation matrices, which becomes ineffective when dealing with large numbers of parameters or training samples.In contrast to previous approaches, prompt-based CL methods freeze the parameters of backbone, and train prompts inserted into the outputs of hidden layer instead, thus providing adaptation to new tasks. Learning to Prompt (L2P) [50] introduces the concept of prompt pool, which selects prompts with the highest similarity for different inputs. Inspired by complementary learning systems, DualPrompt [49] designs two types of prompts to learn general and specialized knowledge respectively. However, L2P and DualPrompt are inseparable from the step of prompt selection from a prompt pool, which cannot be optimized by backpropagation. For this reason, COntinual Decomposed Attention-based Prompting (CODA-Prompt) [41] introduces prompt decomposition, which decomposes a task-specific prompt into a linear combination of multiple prompts. S-Prompt [47] advocates learning prompts independently according to domains to avoid confusion between new and old knowledge. Language Guidance for prompt-based Continual Learning (LGCL) [18] suggests using natural language prompts to guide the model, while maintaining semantics of prompts also means that the prompt templates are difficult to optimize. Hierarchical Decomposition (HiDe-)Prompt [45] decomposes the objective of prompt learning into task-identity inference, within-task prediction and task-adaptive prediction, which essentially introduces a regularization strategy for prompt-based continual learning.Prompt generation methods train a prompt generator to generate task-specific prompts instead of training the prompts directly. Domain-Adaptive Prompt (DAP) [17] focuses on designing a prompt generator for continual learning, which has better domain adaptability and does not rely on a prompt pool. However, it lacks finegrained guidance for the training process of its prompt generator. Personalized Federated learning framework of client-specific Prompt Generation (pFedPG) [55] propose a personalized prompt generation method in federated learning scenarios, which is fundamentally different from the continual learning settings we consider. Customized Prompts via Language models (CuPL) [35] generates customized prompt for each class of images in the dataset through GPT-3 [3] , while the problem it encounters is the limited control over the generated prompts.Under the perspective of probability prompt learning, deterministic prompts can be viewed as the point estimates of a prompt distribution, which poses challenges in representing the diversity and uncertainty inherently in prompts. PROmpt Distribution leArning (ProDA) [28] designs the first probability model for prompts to estimate the prompt distribution and therefore capture the variability of prompt representations. Prompt Learning with Optimal Transport (PLOT) [6] defines both features and prompt information as discrete distributions under the Dirac measure, and attempts to guide the optimization of prompts through optimal transport theory [44] . Bayesian Prompt Learning (BPL) [7] frame prompt learning from the Bayesian perspective and regularize the prompt space to reduce overfitting on seen prompts. Although these methods also assume prompts as distributions, they are neither continual learning methods nor prompting generation methods. Therefore, GPLS is fundamentally different from the above methods. 
method 3.1 background: 
 Prompt Distribution Learning (ProDA) [28] is the first method in probability prompt learning, which learns an optimal prompt distribution 𝑝 (𝑷 ) targeting at minimizing the empirical lossEQUATIONwhere 𝑷 is the prompt. 𝜽 is the normalized embeddings of the text, which is determined by the prompt distribution 𝑝 (𝑷 ) and the text encoder 𝑔(•). 𝑿 and 𝒀 are the input images and labels respectively. Bayesian Prompt Learning (BPL) [7] assumes that the prompt distribution consists of a set of fixed prompts and residualsEQUATIONHere 𝑷 𝑐 is the fixed prompt for class 𝑐, and 𝑝 (𝒓 |𝑿 ) is denoted as the real posterior distribution for 𝒓. Variational posterior is introduced to approximate the residual distribution, thereby obtaining a lower bound on log-likelihoodEQUATIONAside from the differences in an overall perspective mentioned in Section 2.4, we can also outline three detailed distinctions. (1) We define the prompt distribution by considering prompts as variables that follow a distribution, rather than decompose it into a set of deterministic prompts combined with residual distribution [7] . (2) We train an encoder to generate prompts through a novel variational loss related to Mahalanobis distance, rather than optimize the prompts themselves. (3) We encode task and feature information into task-specific prompts in latent space with the reparameterization technique, rather than training class-specific prompts.In this study, our goal is to generate better prompts 𝑷 for classification tasks, which is different from previous prompt optimization methods. Instead of training the prompts directly, we focus on training a prompt generator to generate better prompts.To achieve this goal, we define the prompts as variables that follow a distribution from the perspective of probability prompt learning [28] , which offers an effective way to describe the uncertainty of the generated prompts. We denote the prior probability density function of prompt distribution as 𝑝 (𝑷 ), where 𝑷 can be seen as a random variable and its value is an observation. In the training process, both the input data 𝑿 and labels 𝒀 are known. The target is to generate a better prompt 𝑷 with the help of 𝑿 and 𝒀 , which can be described as solving a posterior probability density function 𝑝 (𝑷 |𝑿, 𝒀 ).In practice, the value of 𝑷 can be determined by optimizing a generator with 𝑿 and 𝒀 when freezing the parameters of backbone, but it is still difficult to know the distribution 𝑝 (𝑷 |𝑿, 𝒀 ) exactly. There are three main reasons for this: (1) Although a specific 𝑷 can be obtained through optimization, the distribution 𝑝 (𝑷 |𝑿, 𝒀 ) remains unknown because the value of 𝑷 is merely an observation. (2) Even though multiple observations of 𝑷 can be collected In the training stage, the key is optimized through L 𝑚𝑎𝑡𝑐ℎ . In the testing stage, task ID is predicted through the maximum similarity between the trainable key and CLS. Then, we establish a set of tokens for each task and input it into an encoder to obtain the mean 𝝁 and covariance 𝚺 of the prompt distribution. Another reparameterization variable 𝝐 is obtained by normalizing and sampling the inputs of transformer blocks. Finally, the prompts are obtained through reparameterization technique and inserted into the inputs of transformer blocks.through repeated experiments, they do not represent the ground truth. This is significantly different from the scenarios considered by variational auto-encoder (VAE) [19] and generative adversarial network (GAN) [12] , as these methods assume that the observed samples are all real. (3) Although we have defined a symbol for the prior distribution, we do not know the prior information of the prompt distribution. (e.g. The theoretical distribution it follows.) Moreover, since the observed values of 𝑷 are not the ground truth, it is inappropriate to estimate the prior 𝑝 (𝑷 ) through maximum likelihood estimation (MLE) or the posterier 𝑝 (𝑷 |𝑿, 𝒀 ) through maximum a posteriori (MAP).From the above analysis, we can summarize that the challenge in estimating the posterior distribution of generated prompts accurately lies in the absence of prior information and observations representing ground truth. To address this issue, we introduce a variational distribution 𝑞(𝑷 ) to approximate the real posterior 𝑝 (𝑷 |𝑿, 𝒀 ). Since the variational distribution is not real, it can be assumed to follow a certain theoretical distribution.We hope to find an optimal variational distribution that can be expressed as minimizing the Kullback-Leibler (KL) divergence between 𝑞(𝑷 ) and 𝑝 (𝑷 |𝑿, 𝒀 ), namelyEQUATION)Here 𝒬 is the possible value space for variational distribution 𝑞(𝑷 ). However, we cannot assume to know 𝒀 in the testing phase, thus Eq.( 4) is not always valid. To address this problem, we regard prompts as the variable in latent space and expand the logarithmic likelihood probability at the end of the classifier intoEQUATIONAfter further derivation, we find that the log-likelihood log 𝑝 (𝒀 |𝑿 ) with prompt as the latent variable can be expressed as the sum of evidence lower bound (ELBO) and KL divergence, namely log 𝑝 (𝒀 |𝑿 ) = 𝐸𝐿𝐵𝑂 (𝑞) + 𝐾𝐿(𝑞(𝑷 )∥𝑝 (𝑷 |𝑿, 𝒀 )).Here the specific expressions for ELBO and KL divergence are as followsEQUATIONEQUATION)The proof is available at supplementary materials. Since KL divergence is non negative, we can obtain log 𝑝 (𝒀 |𝑋 ) ≥ 𝐸𝐿𝐵𝑂 (𝑞).Since log 𝑝 (𝒀 |𝑿 ) in Eq.( 6) is an observed value, we can transform the objective function in Eq.( 4) to maximizing ELBO as arg maxEQUATIONwhere log 𝑝 (𝑷, 𝒀 |𝑿 ) is the logarithmic likelihood, which can be directly obtained. log 𝑞(𝑷 ) is the logarithmic variational distribution we are concerned with.In Section 3.3, we have derived an implicit formulations for the ELBO in prompt generation. Next, we attempt to further provide its explicit expression. In Eq.( 9), log 𝑝 (𝑷, 𝒀 |𝑿 ) is computable since it has the following formEQUATIONwhere 𝑾 𝑦 and 𝑾 𝑐 are the weight slices of the last linear layer related to class 𝑦 and 𝑐 respectively. 𝑓 (𝑿, 𝑷 ) is the input of the last linear layer. Prompt 𝑷 is generated by encoder 𝜓 (•). In addition,exp(•) 𝐶 𝑐=1 exp(•)is the function of softmax, namely 𝑝 (𝑷, 𝒀 |𝑿 ) = softmax 𝑾 ⊤ 𝑦 𝑓 (𝑿, 𝑷 ) . Through Monte Carlo estimates of expectations [19] , we can obtain the first term of ELBO asEQUATIONThe prompt in each layer is only generated once in each iteration, thus 𝑀 = 1. Since 𝑞(𝑷 ) is not the real distribution of 𝑷 , but an approximate distribution. We can assume that 𝑞(𝑷 ) is a multivariate Gaussian distribution with mean 𝝁 and covariance matrix 𝚺 respectively. Its probability density function can be expressed asEQUATION)Here 𝑘 is the length of the generated prompt, and |𝚺| represents the determinant of 𝚺. We expand the expectation E 𝑞 (𝑷 ) [log 𝑞(𝑷 )] into the integral form and further obtainEQUATION)The proof is presented in supplementary materials. Here (𝑷 -𝝁) T 𝚺 -1 (𝑷 -𝝁) can be regarded as the square of Mahalanobis distance [51, 53] between 𝑷 and 𝝁 from a metric learning perspective. Since prompt length 𝑘 is a predetermined constant, the first term in Eq.( 13) can be ignored during the optimization process. Next, we regard the negative ELBO as the loss function, which can be written in the specific form asEQUATION)We also refer to it as variational loss. Intuitively, the target of Eq.( 14) is to obtain a better variational distribution for prompts. In practical terms, we optimize an encoder that generates task-specific prompts.To ensure that the inverse of the covariance matrix can always be obtained, we perform singular value decomposition (SVD) on 𝚺 and obtain 𝚺 = 𝑼 𝚲𝑽 ⊤ . Here 𝑼 and 𝑽 are left and right singular matrix respectively, and 𝚲 is a diagonal matrix with sorted singular values 𝜆 𝑖 , 𝜆 2 , • • • , 𝜆 𝑘 along its diagonal. Therefore, the inverse matrix of 𝚺 can be expressed asEQUATIONbecause 𝚺𝚺 -1 = 𝑼 𝚲𝑽 ⊤ 𝑽 𝚲 -1 𝑼 ⊤ = 𝑰 . Note that 𝑽 and 𝑼 are both orthogonal matrix according to the properties of SVD. In addition, since covariance matrix 𝚺 is a positive semidefinite matrix, its determinant is non negative. Then |𝚺| equals the product of singular values for the reason that|𝑼 𝚲𝑽 ⊤ | = |𝑼 | |𝚲| |𝑽 ⊤ | = |𝚲|, namely |𝚺| = |𝚲| = 𝜆 1 • 𝜆 2 • . . . • 𝜆 𝑘 .We denote the second and last term in Eq.( 13) as L 𝑛𝑙𝑜𝑔_𝑐𝑜𝑣 and L 𝑛𝑒𝑥𝑝_𝑀𝑎ℎ respectively, which are the abbreviations of negative logarithmic covariance and negative expected (square of) Mahalanobis distance. Their specific forms can be expressed asEQUATIONEQUATIONThus, the final objective function can be expressed asEQUATIONwhere L 𝑠𝑜 𝑓 𝑡𝑚𝑎𝑥 = -E 𝑞 (𝑷 ) [log 𝑝 (𝑷, 𝒀 |𝑿 )] is the negative value of Eq (11) . L 𝑚𝑎𝑡𝑐ℎ is the matching loss applied in L2P [50] , Dual-Prompt [49] and DAP [17] , which optimizes a query for predicting task ID in class-incremental continual learning.Through the above derivation process, we have obtained the explicit expressions of L 𝑛𝑙𝑜𝑔_𝑐𝑜𝑣 and L 𝑛𝑒𝑥𝑝_𝑀𝑎ℎ . Their formulations possess distinct geometric interpretations in the contexts of optimization and metric learning. Specifically, during the backpropagation procedure, the decrease of L 𝑛𝑒𝑥𝑝_𝑀𝑎ℎ corresponds to an elevation in the expectation of squared Mahalanobis distance between the generated prompt and the population moment. When prompt 𝑷 is allowed to be distant from the population moment in the training process, there is a propensity for 𝑷 to explore diverse prompts in a wider area in high-dimensional space rather than being confined to a limited region (e.g. around the saddle point). Therefore, the generated prompts are more likely to achieve better domain adaptability. On the other hand, L 𝑛𝑙𝑜𝑔_𝑐𝑜𝑣 is characterized by the product of eigenvalues, which serves to prevent excessive scaling in the Mahalanobis distance. We explain excess and scaling separately. (1) A small singular value 𝜆 𝑖 will result in a large value of -1 2 log 𝜆 𝑖 , and further increase the total loss. (2) Mahalanobis distance can obtained from Euclidean distance after rotation and scaling [51, 53] . When covariance matrix 𝚺 is an identity matrix, Mahalanobis distance degenerates into Euclidean distance.The prompt in GPLS is generated by reparameterization technique [19] , which can be expressed asEQUATIONwhich ensures that generated prompts 𝑷 follow the similar distribution with 𝝐. We build a sampler to extract the information of hidden layer outputs that generated prompts are inserted into to obtain 𝝐.The mean and covariance matrix of the generated prompts are provided by a trainable encoder 𝜓 (•), namely Here E is the possible value space for the input of the encoder. 𝑬 𝑡 is composed of a set of tokens containing task information, which is similar to the text embedding converted from natural language. We establish a key corresponding to each 𝑬 𝑡 , and train it through a query-key matching formEQUATION)EQUATION)Here 𝑲 𝑡 is the learnable key at task 𝑡, which matchs the CLS of last transformer layer as the query to predict the task ID in testing stage under class-increment settings.Next, we will illustrate how the generated prompts are applied to a Vision Transformer (ViT) backbone. A typical ViT architecture comprises a patch embedding layer along with several transformer layers. When receiving an input image 𝑿 ∈ R 𝑑 𝐻 ×𝑑 𝑊 ×𝑑 𝐶 , the patch embedding layer initially divides the image into a sequence of flattened patches𝑿 𝑝𝑎𝑡 = 𝒙 𝑝𝑎𝑡,1 , 𝒙 𝑝𝑎𝑡,2 , • • • , 𝒙 𝑝𝑎𝑡,𝑑 𝑁 ∈ R 𝑑 𝑁 × 𝑑 2 𝑃 ×𝑑 𝐶 .Here 𝑑 𝐻 and 𝑑 𝑊 are the height and width of an image, and 𝑑 𝐶 is the number of input channels. 𝑑 𝑃 × 𝑑 𝑃 is the resolution of each patch. The quantity of patches can be computed as𝑑 𝑁 = ⌊ 𝑑 𝐻 𝑑 𝑃 ⌋ × ⌊ 𝑑 𝑊 𝑑 𝑃 ⌋.The input of the first transformer layer can be represented asEQUATION)where 𝐶𝐿𝑆 is a learnable class token vector prepended to the sequence of feature patches. Prompt 𝑷 is inserted between class token and patches. 𝑾 0 𝑒𝑚𝑏 ∈ R (𝑑 2 𝑃 ×𝑑 𝐶 ) ×𝑑 𝐷 is a linear transformation projecting 𝑿 0 𝑝𝑎𝑡 into an embedding space, where 𝑑 𝐷 is the embedding dimension. 𝑿 𝑝𝑜𝑠 represents the position embedding. The output of the embedding layer serves as the input for the initial transformer layer. The output of each transformer block is subsequently forwarded to the next transformer block throughEQUATIONEach transformer layer comprises a multi-head self-attention (MHSA) module and a feed-forward neural network (FFN) module. The output of each transformer block comprises a tensor 𝑷 (𝑙 ) at the corresponding position of 𝑷 𝑙 -1 . However, we adopt the prompt 𝑷 𝑙 generated by an encoder for the next layer instead of retaining 𝑷 (𝑙 ) . Namely, [𝐶𝐿𝑆 𝑙 , 𝑷 𝑙 , 𝒁 𝑙 ] is served as the input of next transformer block. 
experiments 4.1 experimental setup: 
 Benchmarks: We conduct experiments on various continual learning datasets, including Split CIFAR-100 [21] , Split DomainNet [34] and Split Pets [33] , Split CropDiseases [31] and Split Eu-roSAT [14] . Split CIFAR-100 contains 60,000 RGB images over 100 Baselines: We compare our GPLS with seven prompt-based continual learning methods, including L2P [50] , DualPrompt [49] , CODA-Prompt [41] , S-Prompt [47] ,LGCL [18] , HiDe-Prompt [45] and DAP [17] . In addition, three representative replay-based approaches including BiC [52] , DER++ [4] and DyTox [9] , two representative regularization-based approaches including EWC [20] and LwF [24] and one dynamic expansion approach ESN [48] adopting ViT as backbone also be considered. All these baselines are continual learning methods designed for class-incremental scenario that task IDs are not available during the testing process. Implementation: We follow the similar implementations as previous work [17] . Specifically, we adopt ViT-B/16 [8] pre-trained on ImageNet as the backbone and Adam (𝛽 1 =0.9, 𝛽 2 = 0.9) with learning rate of 0.01 as the optimizer. All the input images are resized to resolution of 224 × 224, normalized range from 0 to 1 and packaged into 128 samples per batch. We utilize an encoder to generate prompts with a fixed length of 10 at each time. To save computational costs, we adopt the simplest one layer MLP encoder [19] .Evaluation Metrics: We repeat each experiment over 3 times with different random seeds and report their average values with standard errors using three familiar metrics in CL. The experimental results of GPLS and other continual learning baselines on Split CIFAR-100 and Split DomainNet are shown in Table 1 . Note that we take Avg Acc as the primary metric and Forgetting as the secondary indicator. As the first prompt generation method in continual learning, DAP achieved unprecedented performance on Split CIFAR-100 and Split DomainNet by designing the architecture of a prompt generator. Nevertheless, our GPLS adopt an encoder to encode prompts in the latent space and guide the training of encoder through variational Bayesian theory, further improving the performance of prompt generation methods. Specifically, GPLS exhibits the Avg Acc advantages of 2.17% and 6.62% on Split CIFAR-100 and Split DomainNet respectively compared to DAP, which also decreases the forgetting rate by 1.16% and 1.74% on these two datasets.To demonstrate the excellent domain adaptability of GPLS, we also conduct experiments on Split Pets, Split EuroSAT, and Split CropDiseases, which belong to the fields of animals, aviation, and agriculture respectively. As shown in Table 2 , although DAP has achieved high accuracy on these three datasets, we can still further improve performance to state-of-the-art level. In terms of the most important indicator Avg Acc, GPLS are 3.51%, 0.40%, and 0.96% ahead of DAP on Split Pets, Split EuroSAT and Split CropDiseases respectively. Notably, improving performance on the latter two datasets is challenging due to the already elevated accuracy achieved by DAP. In summary, GPLS achieves consistently better accuracy than baselines, which can be attributed to the latent space theory, encoder design, and variational objective function.Ablation on variables for generating prompts. From a holistic perspective, the prompts in GPLS are produced by an encoder. However, a more detailed perspective on component level reveals that GPLS is determined by the collaborative influence of three variables including 𝝁, 𝚺 and 𝝐. Table 3 shows the experimental results after removing each variable separately. (1) Firstly, removing 𝝁 and generating 𝑷 through 𝚺 ⊙ 𝝐 leads to a slight decrease in the overall performance, indicating that the population moment plays an effective role in prompt generation, but its impact is relatively small. (2) Secondly, we remove 𝚺 and construct 𝑷 through 𝝁 + 𝝐. The Avg Acc and Lrn Acc of GPLS exhibit significant declines, and the forgetting rate escalates rapidly. These phenomenon indicates that the overall training framework has suffered from severe catastrophic forgetting. Therefore, 𝚺 plays a more important role in domain adaptation for prompt generation. In addition, L 𝑛𝑙𝑜𝑔_𝑐𝑜𝑣 and L 𝑛𝑒𝑥𝑝_𝑀𝑎ℎ are both related to covariance matrix 𝚺, which indicates that 𝚺 will directly affect the total loss. (3) Thirdly, we ablate 𝝐 and utilize 𝝁 + 𝚺 to generate prompts. The results show that GPLS also experience a decrease in accuracy and an increase in forgetting, which also verifies the effectiveness of 𝝐. Ablation on ELBO constraint. We also conducted ablation experiments on L 𝑛𝑒𝑥𝑝_𝑀𝑎ℎ and L 𝑛𝑙𝑜𝑔_𝑐𝑜𝑣 . The results in Table 4 show that removing any part of the constraints will result in inferior accuracy and higher forgetting. Specifically, deleting L 𝑛𝑒𝑥𝑝_𝑀𝑎ℎ and L 𝑛𝑙𝑜𝑔_𝑐𝑜𝑣 would result in a decrease of 1.15% and 0.66% in Avg Acc respectively, which also show that L 𝑛𝑒𝑥𝑝_𝑀𝑎ℎ plays a more important role in the sense of improving performance compared to L 𝑛𝑙𝑜𝑔_𝑐𝑜𝑣 .Analysis on prompt length 𝑘. To investigate the impact of variation on prompt length 𝑘 for model performance, we set the value of 𝑘 as {1, 3, 5, 8, 10, 15, 20, 30, 50} successively. Table 4 (a) reports Avg Acc and Forgetting when GPLS are performed on Split CIFAR-100. We observed that when the prompt length equals 10, the accuracy and forgetting reach the optimal level simultaneously. Analysis on balancing coefficient 𝛼. The hyperparameter 𝛼 is applied to balance the importance between matching loss and our variational loss. We change its value as {0.1, 0.2, 0.25, 0.3, 0.5, 0.7} successively. As illustrated in Table 4 (b), our method achieves the best performance when 𝛼 = 0.25. Figure 5 : T-SNE visualizations of generated prompts for 10 tasks on Split CIFAR-100. Note that the prompt distribution in GPLS refers to all possible prompts following the prompt distribution, and one generated prompt can be viewed as an observation. In addition, prompts are not utilized to predict classification labels like logits, therefore overlapping parts under different tasks are allowed due to the varying levels of similarity between these tasks.Since both DAP and GPLS are prompt generation methods rather than prompt optimization methods, we compare the prompts generated by our GPLS with those generated by DAP in Figure 5 . Although DAP demonstrates effective domain adaptability through prompt generation, its T-SNE plot reveals challenges in identifying distinct boundaries between the prompts it generates and lacks a unique data center. In contrast, prompts generated by GPLS exhibit a discernible center across various tasks, which indicates that GPLS can effectively identify the commonalities between different tasks. Moreover, prompts in GPLS display clearer boundaries between different tasks, which indicates that GPLS can clearly understand the differences between different tasks and has better domain adaptability. 
conclusion: 
 In this paper, we analyze that the trainable parameters in prompt generation methods have the property of localization, centralization, and indirectness, which poses challenges in achieving higher performance. In addition, we note that the prompts generated by a prompt generator are uncertain and exert an implicit influence on classification results, resembling the concept of latent variables in Bayesian learning. Motivated by this insight, we propose a method called generating prompts in latent space for rehearsal-free continual learning based on variational Bayesian theory. We design a variational encoder which encodes task information and feature representation into prompts in latent space with reparameterization technique. We insert the prompts into the inputs of transformer blocks, and train the encoder by minimizing the negative evidence lower bound. We conduct extensive experiments to verify the effectiveness of our GPLS, and the results demonstrate that GPLS consistently achieves the state-of-the-art performance on five continual learning benchmarks. 
 
 Appendix:  


Decision: The strengths of this paper include a generally well-structured presentation and novel ideas. However, some weaknesses need to be addressed: the presentation of technical details needs improvement, as pointed out by all the reviewers. This includes aspects such as probabilistic modeling, the derivation/formulation of ELBO, and the model design related to the task ID. The authors should address these minor but essential issues in the next version.

Reviews: review: paper_summary: paper_summary: The paper introduces a novel method called Generating Prompts in Latent Space (GPLS) for rehearsal-free continual learning. The core idea is to enhance prompt-based continual learning by treating prompts as latent variables, thereby capturing the uncertainty in prompt generation. GPLS uses a trainable encoder to encode task and feature information into prompts using a reparameterization technique. This encoder generates prompts in the latent space, making it more adaptable to different tasks. The training process is guided by a variational loss function derived from the Evidence Lower Bound (ELBO), specifically related to Mahalanobis distance. Extensive experiments demonstrate that GPLS achieves state-of-the-art performance on various continual learning benchmarks, including Split CIFAR-100 and Split DomainNet.
strength_weakness: strength_weakness: - The paper provides a solid theoretical foundation by leveraging variational inference and the Evidence Lower Bound (ELBO). This approach is mathematically sound and aligns with the principles of probabilistic modeling, enhancing the credibility and robustness of the proposed method.
- The method is technically sound, with a well-defined problem formulation and solution. The use of a trainable encoder and the reparameterization technique ensures that the generated prompts are effective and adaptable.
- The paper is well-organized and clearly written.
limitations: limitations: - The paper adopts a simple one-layer MLP encoder to generate prompts, which may not fully capture the complex interactions and dependencies in high-dimensional data. In other words, the prompt generation process might be overly simplistic, potentially making the proposed method not fundamentally different from directly optimizing the prompts.
- The paper does not thoroughly address the computational overhead and model size comparison. Detailed analysis and comparisons of computational requirements and scalability with other state-of-the-art methods are lacking.
- The method still relies on predicting task IDs to generate prompts, which is time-consuming and makes the approach similar to prompt selection strategies. This reliance reduces the distinction between prompt generation and prompt selection methods, potentially diminishing the novelty of the proposed approach.
paper_summary: paper_summary: This paper introduces Generating Prompts in Latent Space (GPLS), which considers prompts as latent variables to account for the uncertainty of prompt generation and aligns with the fact that prompts are inserted into the hidden layer outputs and exert an implicit influence on classification. GPLS adopts a trainable encoder to encode task and feature information into prompts with reparameterization technique, and provides refined and targeted guidance for the training process through the evidence lower bound (ELBO) related to Mahalanobis distance.
strength_weakness: strength_weakness: 1. The paper is well organized
2. The formulas and symbols in the article are explained in detail
limitations: limitations: 1. On line 447 of the paper, the authors assume that q(P) is a multivariate Gaussian distribution. How reasonable is this assumption? If q(P) does not follow a Gaussian distribution, what would the loss function look like?
2. How can it be ensured that GPLS can still generate correct prompts for old task samples as new tasks are updated?
3. Sections 3.3 and 3.4 are almost common derivations in variational inference. Could the authors explain in detail what the main contribution of this part is?
paper_summary: paper_summary: This paper tries to provide guidance in training the prompt generator in prompt-based continual learning methods, and propose Generating Prompts in Latent Space (GPLS). utilizes an encoder to encode task and feature information into prompts and provides enhanced, targeted guidance for the training process through variational inference. The experimental results show that the GPLS prompts show clearer boundaries between different tasks, enable a clear understanding of the differences between tasks, and have better domain adaptation.
strength_weakness: strength_weakness: This paper is well organized and easy to follow. The experimental results are good.
limitations: limitations: 1. The authors claim their main focus is “…, existing methods… neglecting the importance of providing effective guidance for training the generator”. However, they have not illustrated this “neglected guidance” problem, and there is no experiment to reflect this point.
2. How do you convert Problem (4) into Problem (9)? Typically, the ELBO is one component of a problem. This problem can usually be split into two distinct parts: one is ELBO and the other is the KL divergence. Taking Equation (6) as an example, we can divide Problem (6) into two separate parts: the ELBO(q) and the KL divergence. Maximizing ELBO(q) equals minimizing the log probability, logp(Y|X). Looking at it this way, the authors suggest that Problem (4) is equivalent to Problem (6). However, I believe this equivalency is either incorrect or somewhat unusual. Please provide a more detailed analysis of this interpretation.
3. How do you get or define the Eq.(9) of $p(P,Y|X)$. In our common sense, $p(P, Y|X)$ informally means: inferring $P$ and $Y$, given $X$. However, the authors consider $P$ and $X$ as inputs, which means inferring $Y$, given $P$ and $X$. It seems unusual or possibly incorrect, please provide a thorough analysis of this. A similar issue also exists in Eq.(11).
4. How to solve the forgetting of the prompt-generator’s forgetting? The main concern of continual learning is to alleviate the forgetting problem of the trainable network, but there is no strategy for this point in the method section after carefully reviewing it.
5. Typos: (1) “Where …” after the equation should not be capitalized, it is a clause of one full sentence. (2) there is “-” left of the KL in the Eq.(7) of Supplementary Material; (3) $d_D$ in Line 612 is undefined and wrong used; (4) suggest replacing “Figure 4a” with “Figure 4(a)” in Line 811.

Please clarify the mentioned concerns, as it is important for the reviewer to update the final decision.
paper_summary: paper_summary: This paper proposes GPLS which generates prompts by an encoder with specific tsak-level tokens and feature information. The paper is well organized with good quality.
strength_weakness: strength_weakness: The proposed method is well designed with good results.
limitations: limitations: As we know, DAP is very sensitive to the batch-wise or  instance-wise match setups in the evaluation stage.  The key difference between the two setups lies in the predicting the task ID for instances in a batch. In the former, a single task ID is predicted for the entire batch, while in the latter, the task IDs are predicted on a per-instance basis. 

The batch-wise setup has a correcting step which is not realistic in real applications. The good performance of the proposed method is still evaluated in the batch-wise setup.  However, the proposed method should also be evaluated in the instance-wise setup which is more challenging without a high matching accuracy.