Vector Quantization Prompting for Continual Learning

Authors: li  jiao
qiuxia  lai
yu  li
qiang  xu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 07:23:54

Abstract: Continual learning requires to overcome catastrophic forgetting when training a single model on a sequence of tasks.   Recent top-performing approaches are prompt-based methods that utilize a set of learnable parameters (i.e., prompts) to encode task knowledge, from which appropriate ones are selected to guide the fixed pre-trained model in generating features tailored to a certain task. However, existing methods rely on predicting prompt identities for prompt selection, where the identity prediction process cannot be optimized with task loss. This limitation leads to sub-optimal prompt selection and inadequate adaptation of pre-trained features for a specific task. Previous efforts have tried to address this by directly generating prompts from input queries instead of selecting from a set of candidates. However, these prompts are continuous, which lack sufficient abstraction for task knowledge representation, making them less effective for continual learning. To address these challenges, we propose VQ-Prompt, a prompt-based continual learning method that incorporates Vector Quantization (VQ) into end-to-end training of a set of discrete prompts.  In this way, VQ-Prompt can optimize the prompt selection process with task loss and meanwhile achieve effective abstraction of task knowledge for continual learning. Extensive experiments show that VQ-Prompt outperforms state-of-the-art continual learning methods across a variety of benchmarks under the challenging class-incremental setting.

Main paper: 
 introduction: 
 Humans have the remarkable capability to continually acquire and integrate knowledge of new concepts or categories without forgetting old ones, whereas deep learning models struggle with catastrophic forgetting [40] when tasked with learning a sequence of classes [42, 10, 39] . Continual learning aims at addressing catastrophic forgetting in deep neural networks (DNNs) by striking a balance between plasticity for learning new incoming data effectively and stability to retain prior knowledge. Approaches in this field vary: some methods dynamically expand network architectures [64, 29, 58] or reconfigure their internal structures [48, 17, 24] for new tasks. Others penalize the update of crucial parameters from previous tasks [20, 27, 65, 1, 49] or alter parameter update rules to prevent interference across tasks [34, 7, 47, 23] . Additionally, certain methods interleave stored past data with current ones for training [18, 8, 44, 45, 4, 6, 36] . Despite recent advances, continual learning remains an open challenge for DNNs.Recently, prompt-based continual learning has emerged as a promising solution to mitigate catastrophic forgetting in sequential task learning. This approach enhances a pre-trained Vision Trans- former (ViT) [11] with a small set of learnable parameters, known as "prompts". These prompts encapsulate task-specific knowledge, shifting the learning focus from the entire model to the prompts themselves, which guide the pre-trained model in generating task-relevant outputs. During inference, the most suitable prompt containing necessary task knowledge is selected from the prompt pool based on the input image to direct the behavior of the frozen pre-trained model.Current prompt-based methods either involve a key-query matching mechanism to select prompts based on the similarity between the image features and the key parameters paired with prompts [62, 61] , or explicitly predict the prompt indices and perform the selection accordingly [60, 59] . However, the non-differentiable nature of indexing impedes the prompt selection from being optimized end-toend with task loss. This limitation can lead to diminished performance, as inaccurate prompt selection may fail to tailor the pre-trained features for the specific task. Efforts to address this issue include implementing a differentiable prompt selection, such as generating prompts as a weighted sum from the prompt pool [50] , or deriving prompts from the intermediate features of the input image [51, 26] . However, the resulting prompts are continuous, which lack the necessary abstraction to effectively represent the task knowledge essential for guiding the pre-trained model to generate features tailored to a certain task. A concept comparison is shown in Fig. 1 .The assumption that discrete prompts better represent task knowledge than continuous prompts for continual learning can be supported by both theoretical insights from cognitive science and empirical evidence. Discrete prompts mimic the organizational structure of memory and knowledge in the human brain, which is typically understood to consist of discrete units such as concepts and facts [19] . This clear separation of information helps prevent interference among different knowledge domains, and enables models to provide distinct guidance for feature extraction specific to each task. Such knowledge abstraction aligns with categorical perception in human cognition, where sensory inputs are perceived as distinct categories (e.g., colors, phonemes) rather than continuous spectrum [41] . Furthermore, empirical comparisons in §5.2 demonstrate the effectiveness of discrete prompts when optimized end-to-end with task loss (e.g., VQ-Prompt V.S. CODA-P or EvoPrompt). In summary, discrete prompts hold significant promise for improving the continual learning capabilities of models, bringing them more in line with human learning.Optimizing prompts with task loss while preserving their discrete properties as representations of concepts poses a non-trivial challenge. In this paper, we introduce Vector Quantization Prompting (VQ-Prompt) for continual learning, which can optimize prompts using task loss while preserving their discrete characteristics as concept representations. This method involves initially generating a continuous prompt and then replacing it with its nearest match from a predefined prompt pool. To address the non-differentiability inherent in prompt quantization, we apply gradient estimation to propagate task loss to the continuous prompt, while additional vector quantization regularization terms further refine the learning of the prompt pool. To further stabilize task knowledge learning, we use representation statistics to mitigate the classification bias towards previous tasks, thereby enhancing continual learning performance.Our contributions are three-folds: (1) We propose VQ-Prompt, an end-to-end learnable discrete prompting mechanism for continual learning, addressing a critical yet overlooked aspect in the current literature. (2) We leverage gradient estimation to pass the task loss to prompt-related parameters while regularizing the learning of the prompts with vector quantization terms, which facilitates the  
related work: 
 Continual Learning refers to the process where multiple tasks are learned sequentially without forgetting [42, 10, 39] . Generally, continual learning has three scenarios [54] . Task-incremental learning (TIL) learns different classes for each task and assumes having task identities available at test time. Domain-incremental learning (DIL) maintains the same set of classes for different tasks while changing the data distributions across tasks, and task identities are not provided for inference. For Class-incremental learning (CIL), each task involves new classes and all the learned classes are to be classified without task identities available during inference. In this paper, we focus on the more representative and challenging CIL scenario. Numerous efforts have been devoted to alleviating catastrophic forgetting. Architecture-based methods address this by either dynamically expanding network architectures [64, 29, 58] or modifying internal network structures [48, 17, 24] for new tasks. Regularization-based methods focus on limiting updates to vital parameters from earlier tasks [20, 27, 65, 1, 49] , or modifying the rules for parameter updates to reduce task interference [34, 7, 47, 23] . Rehearsal-based methods incorporate previous data with current data during training to mitigate forgetting [18, 8, 44, 45, 4, 6, 36] . Despite recent advances, continual learning remains a challenging and evolving field.Prompt-based Continual Learning Methods. Recently, there has been a surge in methods leveraging prompting techniques from natural language processing (NLP) [28, 30] for continual learning. These methods instruct a frozen pre-trained transformer using learnable prompts that encode task knowledge. During training, prompt selection is either through key-query similarity matching [62] or indicated by task identity [61, 12, 60, 59] . In inference, the appropriate prompt is chosen through similarity matching with key or feature centroids. However, both kinds of prompt selection are non-differentiable, making it challenging to optimize them end-to-end with the task loss, particularly when the gap between the pre-training task and unknown future tasks is large. To address this, CODA-Prompt [50] adopts a soft prompt selection, i.e., generating prompts as a weighted sum from the prompt pool. APG [51] and EvoPrompt [26] learn to derive prompts from intermediate image features. Nevertheless, all three methods generate prompts that are continuous, which lack the necessary abstraction to effectively represent the task knowledge essential for instructing the pre-trained model to produce features tailored to a certain task. In this paper, we present a new prompting framework for continual learning capable of optimizing prompts with task loss while preserving their discrete properties as the representation of task knowledge.Vector Quantization in Representation Learning. Vector Quantization (VQ) is a technique used in signal processing and data compression to represent a set of vectors (data points) with a smaller set of "coding vectors" (CVs). Unsupervised VQ algorithms such as Self-organizing Maps (SOMs) [22] and Neural Gas (NG) networks [38] attempt to obtain a set of CVs that optimally represent the data. Supervised VQ algorithms such as Learning Vector Quantization (LVQ) [21] focus on reducing the misclassification rates by refining decision boundaries between classes. In generative modeling, VQ has been used to learn structured discrete latent spaces in VQ-VAE [55] and VQ-GAN [13] to achieve higher fidelity images. Recently studies have explored combining VQ with continual learning to constrain the feature space, aiming to enhance class separation and retrain prior knowledge across increments [52, 53, 9, 37] . In this paper, instead of utilizing VQ to confine the feature space, we employ VQ to enable end-to-end learning a set of discrete prompts that effectively encode task knowledge in a learning system that evolves over time. Prompt-based Learning is an emerging approach in NLP [32] that involves incorporating extra instructions into pre-trained models to guide their performance on specific tasks. Rather than relying on extensive retraining or task-specific fine-tuning, this technique leverages prompts to shape the behavior of pre-trained models, providing adaptable instructions that helps them handle a wide range of downstream tasks more effectively. In vision-related continual learning, prompting is typically employed with Vision Transformer (ViT) [11] .ViT consists of a sequence of multi-head self-attention (MSA) blocks [56] . For clarity, we take one MSA block as an example to illustrate the prompting. We denote the input query, key, and value of the MSA block as h Q , h K and h V , respectively. Here, h * ∈ R L×D , L is the sequence length, and D is the embedding dimension. The output of the MSA is computed as:EQUATIONwhereW O , W Q m , W K m and W V m are projection matrices, m = 1, • • • , M is the head index, M is the number of heads, and h Q = h K = h V = h for SA.Previous prompt-based continual learning methods mainly implement Prompt Tuning (Pro-T) [28] and Prefix Tuning (Pre-T) [30] . Pro-T prepends the same prompt p ∈ R Lp×D to h Q , h K , and h V . The prompting function of Pro-T is defined as:EQUATIONwhere [•; •] means concatenating along the sequence length dimension. The output dimension is (L p +L)×D. Pre-T splits p along the sequence length dimension into p K , p V ∈ R Lp/2×D , which are prepended to h K and h V , respectively:EQUATIONThe output dimension is the same as that of h. In continual learning, the pre-trained ViT backbone is kept frozen as a general feature extractor, and the prompt parameters p are trained to capture task knowledge. Proper prompts corresponding to the input samples are selected to guide the feature extraction during inference. Following [61, 50, 59] , we adopt Pre-T strategy in our method. 
method: 
 As shown in Fig. 2 , our VQ-Prompt approach begins by constructing a continuous prompt through a soft selection from the prompt pool ( §4.1). The continuous prompt is then quantized to an element in the prompt pool, which is inserted into an MSA block of a frozen pre-trained transformer. This process is made end-to-end trainable through gradient estimation and vector quantization regularization ( §4.2), such that the prompting parameters, namely the keys and the prompt pool could all be optimized using the task loss. In this way, VQ-Prompt can yield a discrete prompt for each input while maintaining end-to-end optimization. To better stabilize task knowledge learning, representation statistics of previously learned classes are employed to mitigate the classification bias ( §4.3).Most previous prompting-based continual learning approaches construct their prompts by selecting from the prompt pool based on key-query similarity [62, 61] or other task identity prediction mechanisms [60, 59] , making the prompt selection process non-differentiable. In our prompt formation, we first generate a continuous prompt by aggregating all the elements in the prompt pool based on the similarity scores between the query and the keys. Specifically, given a query q from the input image, the similarity score is calculated as: An input image is passed through a query function (e.g., a fixed pre-trained ViT) to generate a query q, which is then used to compute similarity scores with prompt keys K. These scores α serve as weights to aggregate elements from the prompt pool P to form a continuous prompt p ′ . This prompt is subsequently quantized to an element within the prompt pool p, and then fed into a specific MSA block of a frozen pre-trained transformer. To ensure differentiability, the prompt quantization process employs gradient estimation and prompt pool regularization. The representation statistics of features from learned classes are used to stabilize task knowledge learning. More details are shown in §4.EQUATIONwhere K ∈ R N×D is the prompt key matrix, q ∈ R D is the query, N is the number of keys, and D is the embedding dimension. Then, the continuous prompt is obtained by:EQUATIONwhere P i ∈ R Lp×D is the i-th element in the prompt pool, and L p is the length of the prompt. Such a prompt formation process is differentiable and can be viewed as a simplified version of CODA-P [50] .Here, we do not learn an extra attention parameter for weighting the query, nor do we increase the number of elements in the prompt pool or the number of keys during sequential task learning.Nearest-neighbour Look-up. The continuous prompt p ′ obtained in Eq. ( 5) is conditioned on a specific instance, i.e., it varies with the input images, making it insufficiently abstract to capture task knowledge effectively. We further perform prompt quantization by performing the nearest neighbour (NN) look-up in the prompt pool P using p ′ . The quantized prompt to be fed to the MSA block is obtained by:EQUATIONSuch a prompt selection pipeline can be viewed as a specific non-linearity that maps the continuous prompt to 1-of-N elements in the prompt pool.Gradient Estimation. Because the arg min operation in Eq. ( 6) is non-differentiable, we use the straight-through estimator [3] to approximate the gradient of p ′ using the gradient of p. Despite its simplicity, this estimator has demonstrated its effectiveness in our experiments. Specifically, in the forward process, the quantized prompt p is passed to the MSA block in the pre-trained transformer.During the backward computation, the gradient of p is transferred unaltered to p ′ , and the optimization of prompt pool P and keys K guided by the similarity scores (c.f., Eq. ( 4)). This gradient estimation is justified, as p and p ′ share the same L p ×D-dimensional space, and the gradient of p provides valuable information on how prompt parameter learning could instruct the transformer features to minimize the cross-entropy (CE) loss during task learning. In this way, each prompt and key element is adjusted according to its relevance to the current learning context, rather than undergoing wholesale changes. This allows for more updates of task-relevant elements without disrupting less relevant ones, thereby maintaining previously acquired knowledge while adapting to new tasks.Vector Quantization (VQ) Regularization. Though the prompt pool P could receive gradients from the task loss through straight-through gradient estimation of mapping from p to p ′ , to enhance the learning of the prompt embedding space, we add an extra VQ objective. This VQ objective uses the L 2 error to move the selected element p of prompt pool towards the continuous prompt p ′ :EQUATIONHere, sg[•] stands for the stop-gradient operation [55] , which constrains its operand to be a nonupdated constant during training.To ensure that the learning processes of the prompt keys K and the continuous prompt p ′ align closely with the characteristics of the element p from the prompt pool P , we further introduce a commitment regularization term. This term is defined mathematically as follows:EQUATION)The incorporation of this commitment loss ensures that the prompt formation process described in §4.1 is optimized to yield prompts to commit to the elements in the prompt pool P , thereby promoting consistency and stability in the prompt learning process.Though prompts effectively capture task knowledge to guide the backbone f (•) in producing instructed representations, the classifier ϕ(•) may develop a bias towards new classes in continual learning scenarios [16, 2] . This bias can adversely affect the learning of the prompts for subsequent tasks.To mitigate this issue and stabilize task knowledge learning, we employ a strategy similar to [59] , which leverages the representation statistics of previously learned classes to correct classifier bias and stabilize prompt learning. Specifically, after completing task t, we calculate the mean µ c and variance σ c for each class c ∈ Y 1:t with the learned prompt parameters and the pre-trained backbone. By modeling each class as a Gaussian distribution, we generate pseudo features through sampling from these distributions. These pseudo features are then used to fine-tune the classifier, thereby mitigating its bias towards recent classes. The balanced classifier could help stabilize task knowledge learning in the prompts, alleviate catastrophic forgetting, and enhance overall continual learning performance.The overall loss function is defined in Eq. ( 9), which extends the task loss L CE with two terms, namely a quantization objective L VQ weighted by λ q , and a commitment term L Commit weighted by λ c .EQUATIONHere, L CE is the cross-entropy loss that supervises the learning of the image classification task, L VQ is the VQ regularization defined in Eq. ( 7) that updates the prompt pool elements to move towards the continuous prompt p ′ , and L Commit is the commitment term defined in Eq. ( 8) that forces the prompt formation process to commit to the prompt pool elements. During training, the pre-trained backbone f (•) is frozen, while the classifier ϕ(•) and prompting parameters K and P are optimized across all the tasks. During task knowledge stabilization, only the classifier ϕ(•) is actively trained. 
experimental setups: 
 Datasets. We consider three representative benchmarks for evaluating CIL. ImageNet-R [14] includes 200-class images that are either hard samples for ImageNet or newly collected data with different styles, thus can serve as a challenging benchmark for continual learning with pre-trained models.In the experiments, we divide it into 5, 10, and 20 disjoint tasks and report the corresponding performance. Split CIFAR-100 randomly splits the original CIFAR-100 [25] into 10 disjoint tasks, each containing 10 classes. Split CUB-200 is built on CUB-200-2011 [57] , a fine-grained classification dataset, by randomly splitting the 200 classes into 10 tasks, where each task contains 20 classes.Baselines. We evaluate our approach against a comprehensive set of baselines to contextualize its performance. Following [50] , we include "Joint Training" as the upper-bound performance, setting a benchmark for optimal results. To establish lower bounds, we employ two sequential learning baselines, denoted as "FT" and "FT++", with the latter refraining from updating the logits of previously learned classes during the training of new tasks.We also consider 5 prompt-based approaches: L2P [62] , DualPrompt [61] , HiDe-Prompt [59] , CODA-Prompt [50] and EvoPrompt [26] , where the last two yield continuous prompts. For L2P, we include its two variations from [50] : "L2P++" and "Deep L2P++". L2P++ uses Pre-T instead of Pro-T and inserts the prompts to the first MSA block, which achieves better performance than the original L2P [33] . Deep L2P++ extends L2P++ by incorporating prompts into the first five MSA blocks.In addition to prompt-based methods, we include a classical regularization-based method LwF [31] , and a rehearsal-based method BiC [63] , providing a more comprehensive overview for evaluation.Evaluation Metrics. We present Final Average Accuracy (FAA) and Cumulative Average Accuracy (CAA) for comparison. FAA refers to the last average accuracy after learning all the tasks, which is equivalent to "Last-Acc" in [67] . CAA is the average of historical FAA values after learning each task, which is equivalent to "Inc-Acc" in [67] . The formal definitions of the metrics are in §A.1.Implementation Details. We follow prior works [62, 61, 50, 59, 67, 26] and use ViT-Base [11] pre-trained with supervised learning on ImageNet-1K [46] or ImageNet-21K [43] as the backbone.The number of keys and prompt elements N is 10. The prompt length L p is 8. The embedding dimension D = 768 which is the same as the feature dimension of ViT-Base.Our method is trained using an AdamW optimizer [35] with an initial learning rate of 0.0025 and a cosine decay schedule. The batch size is 128 for Split CIFAR-100 and Split CUB-200, and 64 for ImageNet-R. The number of epochs is set to be 20 for training on all three datasets. The classifier bias mitigation process described in §4.3 requires ten epochs of training. Each experiment is run on a single NVIDIA GeForce RTX 4090 GPU. More details are presented in §A.2.In this section, we present a comprehensive comparison with established baselines across various datasets and pre-training regimes. The performances of different methods are reported in separate tables due to the varying experimental settings such as the number of tasks and the pre-training dataset, to ensure a fair and accurate comparison.Results on ImageNet-R. Table 1 shows the results across five runs on ImageNet-R with 5-task, 10-task, and 20-task splits using the ViT-Base backbone pre-trained with supervised learning on ImageNet-1K. Our VQ-Prompt consistently outperforms other methods across key metrics such as FAA and CAA for all task splits, including the latest prompt-based method EvoPrompt.Results on Split CIFAR-100. Table 2 presents the results across five runs on CIFAR-100 split into 10 tasks, with the ViT-Base backbone also pre-trained on ImageNet-1K with supervised learning. Our VQ-Prompt achieves superior results compared to other methods using the same pre-training weights.Results on Split CUB-200. Table 3 displays the results on Split CUB-200. Following [67] , we use the ViT-Base backbone pre-trained on ImageNet-21K for this dataset. VQ-Prompt achieves superior or comparable performance compared with all other methods, including SLCA [67] , which trains the entire network without freezing the pre-trained feature extraction backbone. This highlights its potential and efficacy in continual learning for fine-grained classification tasks.Other Pre-training Regimes. Table 4 summarizes the experimental results across three runs on the 10-task ImageNet-R dataset, utilizing different self-supervised pre-training paradigms, namely iBOT-1K [68] and DINO-1K [5] . These results demonstrate that our method consistently outperforms state-of-the-art prompt-based continual learning methods, underscoring its robustness and efficiency in leveraging self-supervised pre-training for continual learning tasks. Specifically, VQ-Prompt shows a greater advantage in CAA than FAA, indicating its superior ability to leverage past knowledge for aiding current tasks, despite a slightly higher degree of forgetting relative to some baselines. This trade-off between adaptation to new tasks and forgetting of previous ones is a common challenge in continual learning. With self-supervised pre-training, our method tends to prioritize adaptability to new tasks to ensure that the model remains relevant and effective in dynamic environments.In this section, we assess the effectiveness of different components illustrated in §4. The experiments are performed on 10-task ImageNet-R with the ViT-Base backbone pre-trained on ImageNet-1K.Effectiveness of VQ Design. Our VQ design (c.f., §4.2) enables end-to-end training of the discrete prompt selection in continual learning. Here, we compare it with an alternative intuition design choice, i.e., rewriting Eq. ( 4) as α = Softmax(Kq/τ ), and reducing the temperature τ of the softmax operation. A lower temperature leads to a "sharper" distribution of α, allowing the prompt formation in Eq. ( 5) to more closely approximate discrete prompt selection during end-to-end training with task loss. This baseline is denoted as "Soft-Prompt". Fig. 3 The rationale behind this is that reducing the temperature makes the softmax operation more sensitive to differences in logits. While this heightened sensitivity is acceptable when the model is confident in its predictions, it can lead to more aggressive prompt choices at the beginning of the training when the model is less fully trained. In contrast, our VQ-Prompt utilizes a standard softmax for prompt formation, substitutes the resulting prompt with the nearest one in the prompt pool, and enables end-to-end learning through gradient estimation and VQ regularization, which proves to be more robust for task knowledge learning compared with Soft-Prompt.Hyperparameters for Prompting. There are two key hyperparameters: i) the size of the prompt pool N that represents the total capacity of the learnable prompts, and ii) the length of a single prompt L p which determines the capacity of a single prompt to encode certain aspects of task knowledge. The total size of the prompts to be prepended to the input of one MSA block is given by N ×L p .Fig. 3 (b) illustrates the impact of varying L p and N on FAA performance. Across different parameter configurations, our method consistently outperforms existing approaches, demonstrating its robustness. Specifically, an excessively small L p value consistently yields sub-optimal results, as indicated by the lower FAA scores across different N values. Increasing N can partially compensate for small L p values, leading to improved performance. In contrast, increasing L p generally enhances performance up to a certain threshold, beyond which an overly large L p may cause knowledge overfitting, as reflected by the stable or slightly declining FAA scores for larger L p values. We selected L p = 8 and N = 10 as our default configuration. This configuration achieves superior results with fewer parameters compared to other prompt-based methods (c.f., § A.3).Our competitive performance is primarily attributed to the use of VQ, which offers several key benefits for prompt-based continual learning. First, VQ enables the encoding of task knowledge into discrete prompts, which provide a more compact representation than continuous prompts. This discrete nature helps in capturing essential task-specific features with the necessary level of abstraction. Second, integrating VQ within the prompt-based framework facilitates end-to-end optimization with task loss, ensuring that the selected prompts are highly relevant to the task at hand, thereby enhancing the task knowledge learning of the prompts. This enables the use of shorter prompts while maintaining strong performance, making our approach more parameter-efficient and effective.Impact of λ q and λ c . To further enhance the learning of prompt-related parameters, we introduce two regularization terms L VQ and L Commit to guide the learning (c.f., §4.2). We investigate the impact of various loss weights, as shown in Eq. ( 9). The outcomes are detailed in Fig. 3 (c ). As can be observed, these two terms can contribute to good performance when assigned with a relatively broad range of values. According to Fig. 3 (c), we set λ q = 0.4 and λ c = 0.1 in all of our experiments.Effectiveness of Classifier Bias Mitigation. The classifier bias mitigation utilizes representation statistics to stabilize task knowledge learning (c.f., §4.3), which can also be applied to other methods.To evaluate its potential advantage, we integrated this component into L2P++. As shown in Table 5 , L2P++ with representation statistics ("L2P++ V2") achieves improved performance over the original L2P++ across all three ImageNet-R settings, but remains inferior to our method. Additionally, we include the results of "VQ-Prompt-S", a simplified version of VQ-Prompt that omits classifier bias mitigation. Notably, VQ-Prompt-S still outperforms other methods such as EvoPrompt, demonstrating the effectiveness of our approach. This indicates that the classifier bias mitigation process can contribute to performance improvements, but is not the sole determinant of the final performance. 
discussion and conclusion: 
 This study focuses on one critical deficiency inherent in current prompt-based continual learning methodologies, specifically the end-to-end optimization of the prompt selection process with task loss while keeping its discrete nature as the representation of task knowledge. Our proposed Vector Quantization Prompting (VQ-Prompt) framework mitigates the challenge by substituting continuous prompts with their nearest counterparts from the prompt pool, thereby enhancing task accuracy through a more aligned and abstract representation of conceptual task knowledge. To overcome the non-differentiability inherent in this process, we employed gradient estimation along with vector quantization regularization terms, which allows for optimizing prompt retrieval with task loss. Representation statistics are utilized to further stabilize task knowledge learning. Extensive experiments in class-incremental scenarios consistently demonstrate VQ-Prompt's superiority over SOTA methods.Limitations and Future Work. One limitation of VQ-Prompt is its dependence on pre-trained models. While these models offer rich initial knowledge, enabling a more mature learning process akin to that of an adult, they also inherit the limitations of the pre-trained data distribution and the high computational costs associated with their use. This challenge is not unique to VQ-Prompt but applies broadly to other continual learning methods that rely on pre-trained models. Another limitation is the absence of constraints in calculating similarity scores and prompt keys, which can result in suboptimal prompt utilization, i.e., some prompts are more frequently selected for samples from different tasks while others are less frequently used. To alleviate this, one possible strategy is to introduce constraints on prompt selection, such as limiting the reuse of prompts that have already been heavily utilized by previous tasks to enhance the diversity and utility of the prompts. We leave a more thorough exploration of such prompt selection constraints for future research. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: Social impacts are discussed at the end of the paper.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: The paper does not release data or models that have a high risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: The paper properly cites the original paper that produced the code package or dataset. The license and terms of use are explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: This paper presents a prompt-based continual learning method to tackle the challenges of knowledge preservation in sequential task learning. Our approach facilitates continual adaptation and learning, thereby contributing to the advancement of intelligent, adaptive, and efficient technologies applicable to various domains, including autonomous vehicles and personalized AI agents. While VQ-Prompt advances class-incremental continual learning, its robustness could be compromised if poisonous samples are introduced after a concept has been learned. One possible mitigation strategy is to implement robust anomaly detection methods to identify and filter out suspicious input samples before they are introduced into the training process.Answer Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: I agree with the unanimous decision of the reviewers to accept the paper. However, there has been some discussion regarding its presentation. I strongly encourage the authors to incorporate the reviewers' feedback and include the new results in the camera-ready version of the paper.

Reviews: review: paper_summary: paper_summary: Prompt-based continual learning has emerged to address catastrophic forgetting in sequential task learning by using a pre-trained Vision Transformer (ViT) enhanced with learnable "prompts." These prompts contain task-specific knowledge and guide the model in generating task-relevant outputs. During inference, the most suitable prompt is selected based on the input image. However, current methods face challenges due to non-differentiable indexing, leading to suboptimal prompt selection and performance issues. Discrete prompts offer better task knowledge representation and prevent interference among different tasks. This paper introduces Vector Quantization Prompting (VQ-Prompt) for continual learning, which optimizes prompts using task loss while maintaining their discrete properties. The method generates a continuous prompt and replaces it with the nearest match from a prompt pool, using gradient estimation to handle non-differentiability and additional regularization to refine the prompt pool. Representation statistics are used to mitigate classification bias towards previous tasks, improving continual learning performance.
strength_weakness: strength_weakness: The performance improvements are meaningful, but not significant - particularly on specific tasks, such as CIFAR-100 and CUB-200 benchmarks. 

The technical contribution of the proposed method is not solid. Indeed I didn't find unique insightful ideas/analyses/novelty that I can get only from this paper.  Most technical components used in this work are somewhat general in vector quantization (e.g., NN look-up and straight-through gradient estimation) and continual learning literature (e.g.,post-adjustment of representation based on statistics). And this proposed method is more like a good combination of vector quantization idea for prompt design during continual learning.

The question of "Why is the vector quantization idea more beneficial for prompt-based continual learning against other baselines" is not sufficiently discussed. It achieves competitive performance with a smaller prompt length than other continuous-prompt methods. However, the paper does not provide any analysis/evidence as to why this happens, but only gives typical ablation studies.


In Table 4, the proposed method seems to have pros and cons for different self-supervised pre-training paradigms. It shows better cumulative accuracy (CAA) for on-par final accuracy (FAA); in my understanding, this means the proposed method shows a higher degree of forgetting compared to baselines, while showing better adaptation performance to new tasks.
questions: questions: -
limitations: limitations: The authors addressed the limitations and broader impacts.
paper_summary: paper_summary: This paper introduces Vector Quantization Prompting (VQ-Prompt), a novel prompt-based method designed to mitigate catastrophic forgetting in the sequential learning scenario of Class Incremental Learning (CIL). VQ-Prompt utilizes Vector Quantization (VQ) to facilitate end-to-end learning with a set of discrete prompts. Initially, it computes the queries for the input images and determines similarity scores between the query and keys, similar to the Learning to Prompt (L2P) approach. This involves using a pretraining image method to generate queries. Subsequently, it aggregates prompts based on these similarity scores, resulting in what is called a continuous prompt. This aggregated prompt is then quantized by selecting the nearest neighbor (NN) from the prompt pool. Then, the pipeline to classify the instances remains constant compared to L2P. To ensure gradient propagation through the key and all prompts, VQ-Prompt employs the straight-through estimator along with similarity scores. Additionally, it incorporates representation statistics to address classification bias on the classifier. The authors demonstrate that VQ-Prompt outperforms state-of-the-art (SOTA) baselines across three datasets: ImageNet-R (5, 10, and 20 tasks), CIFAR-100 (10 tasks), and CUB-200 (10 tasks).
strength_weakness: strength_weakness: 1. Considering Prompt Learning for Continual Learning (CL) assumes that there is a good and generalizable pretraining that could be transferable (tuned) to the different downstream tasks. Moreover, considering the way that the queries are generated, it also presumes that this pretraining can accurately select prompts for all tasks. I would like to see results on more challenging tasks like out-of-domain. Although the authors perform the evaluation in different scenarios, two of them, ImageNet-R and CIFAR-100, are closely related to ImageNet. For instance, ImageNet-R is derived from ImageNet-1K, and also its classes overlap with ImageNet-1K.
2. Additionally, it is worth noting that the authors use a larger pretraining dataset for the experiments in Table 3, where VQ-Prompt achieves nearly the same performance as HiDe-Prompt. This raises the question of whether this is related to the initialization issue mentioned earlier. I would like to see these results with a different initialization to understand this relationship better.
3. There is no consistency in the evaluation of state-of-the-art (SOTA) methods across all tasks (Table 1, Table 2, Table 3), making comparison and analysis challenging. For instance, HiDe-Prompt shows a significant improvement over CODA-P and almost matches the performance of VQ-Prompt. So, why is it not evaluated on the other tasks? I would like to see how HiDe-Prompt performs on the other tasks presented in Table 1 and Table 2.
questions: questions: Please refer to the weaknesses section, I left some questions for you.
limitations: limitations: The authors mention a limitation of this method: scaling up the prompt size does not increase performance. However, this is not necessarily a limitation. This could indicate that the model does not need significant parameters to learn the task because it is relatively easy due to its initial knowledge. (Please see point 1 of the weaknesses).
paper_summary: paper_summary: The representations learned by large pre-trained models have led to many Continual Learning approaches based on these models. Specifically, prompt-based approaches train a small set of learnable parameters (prompts) to guide a fixed pre-trained model for a particular task. One key component in these approaches is the selection of relevant prompts, which helps to specialise the guidance depending on the need for the corresponding input. Most previous works select the relevant prompt based on the similarity of the input and a key in the representation space of the same visual pre-trained model. The key is commonly connected to the prompt via a key-value data structure, making it unfeasible to traceable optimisation sequence and infeasible to optimise with task loss. In this paper, the authors propose Vector Quantization Prompting, which incorporates vector quantisation into an end-to-end training process, including the keys and prompts. By using a look-up NN over a weighted sum of all the prompts and a straight-through estimator to approximate the gradient for the prompts and keys, the authors were able to optimise not only the prompts but also the key when selecting relevant prompts, which, as shown in the results, can help achieve better performance in diverse benchmarks.
strength_weakness: strength_weakness: - One of the authors' motivations is learning discrete prompts, which they even mentioned as a contribution. However, even after reading the explanation between lines 50 and 61, the contribution of having discrete prompts is unclear. I understand the motivation of having concrete concepts represented as discrete vectors for human understanding, but a continual vector can easily represent the same concept and also be distinct categories and even linearly separable.
    - Furthermore, the implementation of discrete prompts in the proposal must be clarified. While I assume it occurs in the NN look-up or the prompt formation, there is a lack of ablation on this point. For instance, how is prompt usage distributed (Are they uniformly distributed, or are some very specialised)? Are there specialised prompts? Is there a discernible relationship between a 'concept' and a prompt?
questions: questions: - In line 195, it is mentioned that this selection allows more updates in relevant prompts without disrupting less relevant ones. This idea is very interesting and crucial in CL. However, some questions remain:
    - Concerning this point, how different is a traditional prompt-based approach from the proposal? In both cases, you only update the “relevant” prompt, but the meaning of relevant changes. Or is there something else?
    - The look-up table selects the most relevant prompt; however, we should be able to compose concepts to learn more abstract ideas. Can this proposal be extended to this?
- There is a second training phase when applying the component described in Section 4.3. How many epochs does this new phase have? Could it be an unfair advantage over other methods? For example, how does L2P behave with this component?
- The experiments done in 5.3 concerning the temperature of alpha suggest that a lower value leads to a sharper distribution, which can lead to a more discrete selection; however, a higher value achieves better performance. How can we read this, as it seems contrary to the need for discrete prompts?
limitations: limitations: The authors present some limitations and concerns regarding the proposal. Another limitation that can be added is the intrinsic limitation in the pre-trained models concerning the limitations of the pre-trained data distribution and the computational cost of running these models.
paper_summary: paper_summary: This paper presents VQ-Prompt, a prompt-based continual learning method using Vector Quantization (VQ) to enhance task knowledge representation and overcome catastrophic forgetting. VQ-Prompt incorporates VQ into the end-to-end training of discrete prompts, optimizing the prompt selection process with task loss and achieving effective task knowledge abstraction. Extensive experiments demonstrate that VQ-Prompt outperforms state-of-the-art methods in various benchmarks, particularly in class-incremental settings.
strength_weakness: strength_weakness: 1. There is no constraint when calculating the similarity score \alpha and prompt key K, which may lead to corrupt prompt learning, i.e., for most test samples from different tasks, the selected prompts are similar, and large parts of prompts are useless.

2. Lack of ablation study on the fine-tuned classifier. Fine-tune classifier is a trick to improve the classifier performance, which can also be applied to other methods. It is unfair to directly compare with other methods.
questions: questions: 1. In Fig. 3(a), temperature $\tau=1$ achieves the best performance, then what about the temperature larger than 1?
limitations: limitations: NA

Comments: title: Thanks for the rebuttal
Rebuttal Response
comment: I thank the authors' answers to my question. After going through all the other reviews and the given rebuttal, I will keep my score to 5 because I'm still concerned the unconstrained prompt selection may be similar, leading to wasted prompts.
I really appreciate the author's effort to address all my concerns. The new results provided by the authors show that the model can work in challenging scenarios for pretraining knowledge. Therefore, I am considering increasing my score.

Best,
Reviewer 7MCa
I thank the author's answers. For now, I will keep my score waiting for comments from the other reviewers.