Black-Box Forgetting

Authors: yusuke  kuwana
yuta  goto
takashi  shibata
irie  go
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 14:26:40

Abstract: Large-scale pre-trained models (PTMs) provide remarkable zero-shot classification capability covering a wide variety of object classes. However, practical applications do not always require the classification of all kinds of objects, and leaving the model capable of recognizing unnecessary classes not only degrades overall accuracy but also leads to operational disadvantages. To mitigate this issue, we explore the selective forgetting problem for PTMs, where the task is to make the model unable to recognize only the specified classes, while maintaining accuracy for the rest. All the existing methods assume ''white-box'' settings, where model information such as architectures, parameters, and gradients is available for training. However, PTMs are often ''black-box,'' where information on such models is unavailable for commercial reasons or social responsibilities. In this paper, we address a novel problem of selective forgetting for black-box models, named Black-Box Forgetting, and propose an approach to the problem. Given that information on the model is unavailable, we optimize the input prompt to decrease the accuracy of specified classes through derivative-free optimization. To avoid difficult high-dimensional optimization while ensuring high forgetting performance, we propose Latent Context Sharing, which introduces common low-dimensional latent components among multiple tokens for the prompt. Experiments on four standard benchmark datasets demonstrate the superiority of our method with reasonable baselines. The code is available at https://github.com/yusukekwn/Black-Box-Forgetting.

Main paper: 
 introduction: 
 Large-scale pre-trained models (PTMs) such as CLIP [Radford et al., 2021] and ALIGN [Jia et al., 2021] have strong capabilities of zero-shot classification for everyday objects. Nevertheless, in practical applications, the classification of all kinds of object classes is rarely required. For example, in an autonomous driving system, it would be sufficient to recognize limited classes of objects such as cars, pedestrians, and traffic signs. We would not need to recognize food, furniture, or animal species. Retaining the classes that do not need to be recognized may decrease overall classification accuracy, as well as cause operational disadvantages such as the waste of computational resources and the risk of information leakage. In this paper, we address the problem of selective forgetting of specified classes [Shibata et al., 2021 , Graves et al., 2021 , Ye et al., 2022 , Tarun et al., 2023] , i.e., tuning a pre-trained model to reduce the classification accuracy for only the specified classes with-38th Conference on Neural Information Processing Systems (NeurIPS 2024). ). The obtained confidence is used to compute the respective loss functions for the classes to be forgotten and the classes to be memorized. (a) For the classes to be forgotten, maximize the entropy of the confidence so that the accuracy is reduced. (b) For the classes to be memorized, minimize the cross-entropy loss to retain the accuracy. These two objective are jointly optimized to tune the learnable text prompt. The gradients of the objective are not available when the model is black-box. We therefore use CMA-ES [Hansen et al., 2003] , a derivative-free optimizer, to learn the text prompt. Instead of directly optimizing the original high-dimensional context (token) embeddings for the prompt, our method learns lower-dimensional latent contexts for mitigating the difficulty of high-dimensional optimization.out affecting the accuracy for the others1 . While selective forgetting of specified classes has long been overlooked [Ye et al., 2022] , a few existing methods have been proposed very recently [Shibata et al., 2021 , Ye et al., 2022 , Tarun et al., 2023] . The seminal work is Learning with Selective Forgetting (LSF) [Shibata et al., 2021] , which has been proposed in the context of continual learning [Kirkpatrick et al., 2017 , Li and Hoiem, 2017 , Aljundi et al., 2018] and uses a special random code called mnemonic code to control the class-wise memorization and forgetting. A similar idea has been proposed to achieve forgetting by learning noise that maximizes the classification error for the classes to be forgotten [Tarun et al., 2023] . An extended version of LSF [Ye et al., 2022] allows forgetting of specified classes as well as recovery of them by temporarily transferring the knowledge of the classes to be forgotten to another network called deposit module.Overall, all the existing methods assume the "white-box" setting, where the complete information of the target model is available for training/tuning, including the model architecture, its parameters, and their gradients. However, major PTMs such as GPT-4V [OpenAI, 2023] are often "black-box," where the model itself or its information is often fully or partially private due to commercial reasons or considerations of social impact. Since the parameters and their gradients are not accessible in such a model, all the existing methods are inapplicable. To the best of our knowledge, selective forgetting methods for black-box models have never been studied to date.In this paper, we address Black-Box Forgetting, i.e., the selective forgetting problem for black-box PTMs, and propose a novel approach to the problem. Given the unavailability of model information, our method, unlike the existing selective forgetting methods, does not optimize network parameters nor utilize the gradients of the parameters; we instead optimize the input textual prompt to decrease the classification accuracy of specified classes to be forgotten in a derivative-free optimization framework. One disadvantage of derivative-free optimization would be that it is not effective nor efficient for high-dimensional problems due to the low convergence rate in high-dimensional spaces [Qian et al., 2016] , and unfortunately, the textual prompt is typically parameterized as a set of high-dimensional vectors in PTMs, e.g., 512-D for each "context" (i.e., learnable token in the prompt) in CLIP ViT-B/16 [Dosovitskiy et al., 2021] . To mitigate this issue, we propose Latent Context Sharing (LCS), a novel parametrization method of the contexts. The core of LCS is to parameterize each context with low-dimensional latent components, which consist of token-specific components and common components among multiple tokens for the prompt. Experimental results on four standard benchmark datasets demonstrate that our method improves zero-shot CLIP and outperforms reasonable baselines based on black-box prompt tuning [Sun et al., 2022b] .The main contributions of this paper are summarized as follows:• We introduce Black-Box Forgetting, a novel problem of selective forgetting for black-box models.• We propose a novel method for Black-Box Forgetting based on derivative-free optimization of learnable text prompt.• We introduce Latent Context Sharing (LCS), a novel parametrization method of contexts for mitigating the difficulty of high-dimensional optimization with derivative-free optimization. 
related work: 
 Machine Unlearning Machine unlearning aims to remove an arbitrary sample from a pre-trained model, i.e., obtaining a model that is identical to the one trained from scratch without that sample [Cao and Yang, 2015 , Golatkar et al., 2021 , Sekhari et al., 2021 , Bourtoule et al., 2021 , Kurmanji et al., 2023 , Guo et al., 2020 , Chen et al., 2019] . Many methods have been proposed, for example, to construct a forgettable model by transforming the learning algorithm into a sum of the training samples [Cao and Yang, 2015] , to achieve forgetting by linear approximation of a nonlinear model [Golatkar et al., 2021] , and to update the model to be closer to / farther from the original model in the retain / forget samples [Kurmanji et al., 2023] . Methods specific to certain learning algorithms such as LDA [Guo et al., 2020] and SVM [Chen et al., 2019] have also been explored. Machine unlearning and Black-Box Forgetting are closely related but different; Machine unlearning aims to remove the influence of specified training samples on the training model, whereas Black-Box Forgetting aims to prevent the recognition of specified classes. Forgetting specified classes has attracted much attention recently in various contexts [Heng and Soh, 2023 , Lu et al., 2024 , Zhang et al., 2024 , Shibata et al., 2021 , Ye et al., 2022] . We in this paper address the black-box setting, which has not yet been explored.Selective Forgetting. Shibata et al. [2021] proposed Learning with Selective Forgetting (LSF), which updates the model for a new task by forgetting only certain classes from the previous task while memorizing the rest of the classes. Golatkar et al. [2020a] introduced a scrubbing method that involves a shift in weight space and addition of noise to the weights to remove information from network weights. They also proposed a forgetting mechanism to linearly approximate the weights that would have been obtained by unlearning [Golatkar et al., 2020b [Golatkar et al., , 2021]] . Tarun et al. [2023] proposed an error-maximization-based method to learn a noise matrix for the class to forget, and the model is updated by training on this noise. Then, fine-tuning on the classes to be memorized to adjust the model weights.Since these methods require the model weights or the gradient of the model parameters, they cannot be applied to black-box models. In this paper, we introduce a new selective forgetting method for black-box models that does not require the model weights or the gradient of the model parameters.Black-Box Learning. Black-Box Tuning (BBT) [Sun et al., 2022b ] is a black-box prompt tuning method for large language models. BBTv2 [Sun et al., 2022a] improves BBT with deep prompt tuning. They achieve accuracy comparable to white-box learning methods in various natural language tasks. BDPL [Diao et al., 2023] fine-tunes a collection of discrete prompts for language models by treating the word choice in the prompt as a reinforcement learning policy.BlackVIP [Oh et al., 2023] , the first black-box learning method for vision-language models, optimizes a generative model that generates visual prompts embedded in images by zeroth-order optimization. LFA [Ouali et al., 2023] extends the capabilities of black-box models by assuming access to pre-computed features from pre-trained backbones. Through a multi-stage procedure, it optimizes a projection layer to enhance alignment between pre-computed image features and class prototypes. Guo et al. [2023] introduced a collaborative black-box tuning (CBBT) for optimizing both the textual prompt and adapting output visual features in black-box vision-language models. The textual prompt is optimized by estimated gradients and the visual adapter is trained through direct supervised learning from the output features.In this study, we focus on textual prompt tuning for the black-box model and introduce Latent Context Sharing (LCS), which improves accuracy while reducing the number of dimensions to be optimized. The overview of the proposed method is illustrated in Fig. 1 . We use the visionlanguage model CLIP [Radford et al., 2021] as the base model and optimize an input prompt for the CLIP text encoder based on the loss that requests reduced accuracy of selected classes. The derivative-free optimization method must be used to optimize a textual prompt for the black-box model where the gradients of its parameters are unavailable. We employ CMA-ES, a widely used evolutionary algorithm for black-box optimization in continuous, because a textual prompt to be optimized is a continuous variable. CMA-ES is a multipoint search algorithm based on a multivariate normal distribution and proceeds the search by iterating (i) sampling candidate solutions, (ii) evaluating the loss values of the candidates, (iii) weighting the candidates based on the loss values, and (iv) updating the mean and covariance matrix of the distribution by using the weighted candidates. Due to the nature of multi-point search, the performance of CMA-ES degrades in high-dimensional problems, typically ten or more dimensions [Ros and Hansen, 2008, Akimoto and Hansen, 2016] . While several extensions have been proposed, e.g., [Ros and Hansen, 2008, Akimoto and Hansen, 2016] , these methods require knowledge of independence among variables, which is not always known. In this paper, we propose a customized extension of CMA-ES to Black-Box Forgetting. In general, when applied to high-dimensional blackbox continuous optimization by CMA-ES, the computational complexity can become a hindrance. The key is reducing the dimensions of the latent variables in the textual prompt while preserving their context representations. 
context parametrization: 
 We discuss two types of context parametrizations: i) Latent Representation with Random Projection for Black-Box Tuning (BBT) [Sun et al., 2022b] ii) Latent Context Sharing. As empirically shown later in Sec. 4.2, the effectiveness of the context parametrization for BBT described above is limited for selective forgetting settings. We propose latent context sharing (LCS), a more efficient context parametrization.Fig. 2c shows the overview of LCS. The key idea is to assume shared parameters among different latent contexts. This inspiration comes from successful word embedding methods; most word embedding methods are trained on the assumption that locally co-occurring words have semantic correlations between them (e.g., [Milkolov et al., 2013 , Pennington et al., 2014 , Devlin et al., 2019] ). This inspires the idea of explicitly modeling semantic correlations between words in a prompt as shared components. We assume that each latent context is composed of unique components (Unique Latent Contexts (ULC)) and common components (Shared Latent Context (SLC)) among multiple latent contexts2 . Then, we optimize each ULC and SLC independently. Each latent context[z i ] is obtained by concatenating SLC [z s ] ∈ R d s and ULC [z u ] i ∈ R d u (i = 1, • • • , m),where m is the number of latent contexts, d s and d u are the dimension of SLC and ULC, respectively. Despite the number of parameters prepared for BBT and LCS is the same (m × d = d s + m × d u ), LCS is possible to significantly reduce the number of optimization dimensions compared to BBT3 , because LCS optimizes each ULC and SLC independently. Compared to assuming that each latent context is completely independent (i.e., using only ULC), providing common components has the substantial advantage of not losing dependencies among multiple tokens for the prompt.Note that, CoCoOp [Zhou et al., 2022a] also introduces an approach that incorporates a shared component in the context of the prompt to improve the generalization in the white-box setting. While CoCoOp learns the network that generates a shared component based on image features, our method directly learns the shared component. The optimization for our black-box forgetting using our method becomes simpler because the number of dimensions for optimization is minimal. As shown in the experimental results in Sec. 4, introducing a shared component is effective, which suggests that optimization of shared components has a impact in our problem settings.We apply different loss functions to the classes to be memorized and the classes to be forgotten. The cross-entropy loss is used for the classes to be memorized to maintain the classification accuracy:EQUATION)where C is the total number of classes, p is confidence of each class obtained by applying the Softmax function to the similarity of each class, which is the output of CLIP, and t is the one-hot vector of the label of an input image.A naive approach to ensuring that selected target classes are forgotten is to reduce the confidence of that class in the input images of the class. In general, however, this naive approach leads to undesirable behavior in the model. For example, if we force the model to forget the class "dog," it may lead to the model always classifying images of "dog" as the class with features similar to "dog," e.g., "cat." Moreover, such an unintentional bias may provide sufficient clues to identify the forgotten classes, and consequently may lead to the risk of information leakage. We, therefore, want to make the classification results for images with the forgotten classes close to random and exclude information about the classes. To this end, we maximize the entropy of confidence for each image of the classes to be forgotten. The loss function for the classes to be forgotten is as follows:EQUATION)where p and C are the same as Eq. (1).To summarize, the final objective that we minimize becomes L = L memorize + L forget . We optimize latent contexts by CMA-ES using the final objective L.Since backpropagation cannot be applied to black-box models, we adopt the CMA-ES (Covariance Matrix Adaptation Evolution Strategy) [Hansen et al., 2003] , which is a derivative-free optimizer for continuous optimization. In the t-th iteration, the CMA-ES samples the λ candidate solutions from a multivariate normal distributionN (m t , σ 2 t • C t ), where m t ∈ R d is the mean vector of the search distribution, σ t ∈ R + is the step-size, C t ∈ R d×d is a covariance matrix.The λ solutions should be evaluated on an objective function f , then the CMA-ES updates the parameters C t , m t and σ t by ranking the λ solutions by function value (cf. [Hansen, 2016] ). 
experiments: 
 We evaluate the class forgetting performance of our method on image classification tasks. We first describe our experimental setup, including the datasets, baselines, implementation details, and evaluation metrics. We then report the main comparative results between our method and the baselines, as well as a series of analyses of our method.Datasets. We use four benchmark datasets, i.e., CIFAR-10, CIFAR-100, CUB-200-2011, and Ima-geNet30 . CIFAR-10 4 and CIFAR-100 5 comprise of a total of 50,000 training images and 10,000 test images [Krizhevsky et al., 2009] . These datasets have 10 and 100 classes, respectively. CUB-200-2011 [Wah et al., 2011] comprises of images of 200 distinct bird species, with 5,994 training images and 5,794 test images. ImageNet30 [Hendrycks et al., 2019 ] is a 30-class subset of the original ImageNet-1k dataset [Deng et al., 2009] (The results on the original ImageNet-1k dataset can also be found in Appendix A.2). It consists of 39,000 training images and 3,000 test images. We conduct experiments in the few-shot condition. We randomly select different k samples of each class from the original training images to construct a k-shot training set and a k-shot validation set. We set k to 16 for CIFAR-10 and ImageNet30, 4 for CIFAR-100, 1 for CUB-200-2011. For testing, we use the original test set. Unless otherwise noted, the first 40% of classes are to be forgotten, while the other classes are to be memorized.Baselines. Black-Box Forgetting has never been studied before, and there is no existing method directly applicable to this problem. So we compare the proposed method with zero-shot CLIP [Radford et al., 2021] , BBT [Sun et al., 2022b] and CBBT [Guo et al., 2023] , which are the reasonable baselines as it is for black-box prompt tuning. We apply the same loss functions as our method to BBT and CBBT (w/o adapter) for comparison. CBBT introduces a method that combines textual prompt tuning and adapting output visual features for black-box vision-language models. However, in this paper, the conditions are such that visual features cannot be obtained, and only the final inference results can be used. So we compared the proposed method with CBBT without using any [Sun et al., 2022b] and CBBT (w/o adapter) [Guo et al., 2023] are the reasonable baselines as these are for black-box prompt tuning. CoOp [Zhou et al., 2022b ] is a white-box method and is included for a reference. Performance is evaluated using the three metrics: the error Err for for the classes to be forgotten, the accuracy Accmem for the classes to be memorized, the harmonic mean H of Err for and Accmem. Higher values mean better performance. adapter. We also apply the same loss functions as our method to CoOp [Zhou et al., 2022b] , a whitebox method, and use the results as reference values under conditions where information on the target model is available.Implementation Details. We use ViT-B/16 [Dosovitskiy et al., 2021] Evaluation Metrics. We use the following three evaluation metrics: (i) Err for is the error for the classes to be forgotten; (ii) Acc mem is the accuracy of the classes to be memorized; (iii) H is the harmonic mean of Err for and Acc mem as in [Shibata et al., 2021] . H gives the overall selective forgetting performance as it is a balance between the forgetting rate for the classes to be forgotten and the classification accuracy for the classes to be memorized. Higher values for all these metrics are desirable. For all the experimental results, including those in the Appendix sections, we report the average performance of the three runs with different random seeds, as well as their standard deviations.Table 1 shows the main comparative results with the baseline methods. Our method improves zeroshot CLIP and outperforms all the baselines on all the datasets.Comparing our method with BBT, these two are comparable in Acc mem . However, ours is significantly better than BBT in Err for and consequently clearly outperforms BBT in H as well for all the datasets; in particular, ours is better than BBT by 9.38% on CIFAR-10. These results suggest that our LCS can optimize the learnable latent contexts more effectively in the black-box setting than BBT, which optimizes all the latent contexts independently.When comparing our method with CBBT, which is the state-of-the-art black-box tuning method, we can see that ours outperforms CBBT on all the datasets. While ours is slightly inferior in Acc mem , Figure 3 : Sensitivity to the number of latent contexts. Results of BBT [Sun et al., 2022b] and Ours for varying number of the latent contexts. We can see that our method shows stable performance within a wide range of the number of latent contexts in contrast BBT.it is significantly better in Err for on all the datasets. These results demonstrate that our method achieves higher selective forgetting performance than CBBT and is a more suitable for the blackbox forgetting task.To clarify the impact of our context parametrization method, LCS, we also evaluate the performance of Ours (w/o LCS), i.e., the case without LCS where each latent context is optimized independently. Although these two are highly comparable in Acc mem , Ours (with LCS) clearly outperforms Ours (w/o LCS) by large margins and consequently shows distinct superiority in H as well. These results prove that adequate selective forgetting performance cannot be achieved without LCS. Interestingly, these evaluations clearly suggest that each latent context is not inherently independent and that the idea of LCS to model dependencies among latent contexts by introducing a common latent component is valid.Finally, we compare our method with the white-box method based on CoOp [Zhou et al., 2022b] ,where the latent contexts are optimized by minimizing the same loss functions as ours through the standard backpropagation process. Although CoOp is better able to increase Err for than Ours, the differences are within the acceptable ranges. For example, the difference in Err for is less than 1% and that in H is less than 2% on CIFAR-10. These results show that even though our method is black-box, it can perform as well as white-box approaches.We investigate the sensitivity of the performance of our method to the number of latent contexts m. Fig. 3 shows Err for , Acc mem and H when the number of latent contexts m is varied on CIFAR-10.As the number of latent contexts m increases, the performance in all the metrics tends to improve. This is natural behavior, as the performance improves as the number of trainable parameters increases. Comparing Ours and BBT, we see that Ours for m = 4 and BBT for m = 16 are almost comparable. This means that BBT needs to optimize about four times the number of dimensions to compete with our LCS, indicating that our LCS provides excellent trade-offs. Furthermore, while BBT suffers a sharp drop in accuracy with decreasing m, our method shows only a small decrease from m = 16 even when m = 1. This suggests that our LCS is robust to the decrease in the number of latent contexts and its significant superiority to BBT for latent context representation.We investigate the sensitivity of our method to d s and d u , which are the number of dimensions of SLC and ULC, respectively. In our LCS, the total number of dimensions d to be optimized is determined as 4 shows the results on the three datasets. First, we can see that for all the datasets, the performance in Err for and H significantly degrades for d s = 0 (i.e., when no SLC is used) than for the other cases, which supports the validity of the core idea of our LCS that introduces the shared components. As expected, the performance is substantially improved by setting the appropriate ratio. This is a natural trade-off: if d s (i.e., the number of dimensions of SLC) is too large, the ability to represent the context is reduced and performance deteriorates; conversely, if d u (i.e., the number of dimensions of ULC) is too large, optimization becomes difficult and performance deteriorates. Meanwhile, our method achieves satisfactory performance in the wide range of d s : d u . Second, when we compare Ours with BBT, Ours is superior to BBT in the wide range of d s : d u . These results justify the strong robustness of our method to d s and d u .d = d s + m × d u (see Sec.CMA-ES [Hansen et al., 2003] is widely used in black-box optimization, but when applied to highdimensional black-box continuous optimization, the computational complexity can become a hindrance. To apply CMA-ES to high-dimensional optimization, modified versions such as Sep-CMA-ES (Sep) [Ros and Hansen, 2008] and VkD-CMA-ES (VkD) [Akimoto and Hansen, 2016] have Figure 5 : Sensitivity to the ratio of the classes to be forgotten. Results on CIFAR-10 when changing the ratio of the classes to be forgotten. Our method is robust to the ratio of the classes to be forgotten compared to baselines.been developed. Sep-CMA-ES realizes the computational complexity O(d) that is linear with respect to the number of dimensions by restricting the covariance matrix to a diagonal matrix. In other words, the solution generation distribution of Sep-CMA-ES does not consider the covariance between variables, but only the variance of each variable. In VkD-CMA-ES, the covariance matrix is expressed as C = D(I d + VV T )D, where D is a diagonal matrix, V is a d × k-dimensional real matrix in which each column is orthogonal, and k is a hyperparameter that determines the degree of freedom of the covariance matrix model. Given the availability of these modified CMA-ESs, one question would be that, can our method still perform better than BBT, even when it is combined with these modified CMA-ESs? Table 2 shows the comparisons between Ours and BBTs with the above two variants of CMA-ES. Ours achieves the best performance in terms of H and Err for for all the datasets. In particular, the two BBT variants show Err for more than 10% lower than our method on CIFAR-100 and ImageNet30. These results show that our LCS surpasses Sep-CMA-ES and VkD-CMA-ES as a context parametrization method in the black-box forgetting task.Fig. 5 shows Err for and H when changing the ratio of the classes to be forgotten r for on CIFAR-10.For BBT, we see a decreasing trend in Err for as the number of classes to be forgotten increases. This suggests that the context representation of BBT is inefficient, making it difficult to forget multiple classes at a time. CBBT shows some robustness, but as with BBT, we see that Err for tends to decrease as the number of classes to be forgotten increases. In contrast, our method does not decrease Err for even if the number of classes to be forgotten increases, which confirms the strong performance of our LCS. In terms of H, compared to the baselines, our method shows high robustness independent of the number of classes to be forgotten and achieves high selective forgetting performance.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: See Sec 4. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
limitations: 
 Our method optimizes the context (token) embeddings of the model through a derivative-free optimization, CMA-ES. That is, we assume that we have access to the context embeddings of the target model. This is a common black-box setting, as similar assumptions have also been made in most of the existing studies [Sun et al., 2022b ,a, Guo et al., 2023] . However, there should be models in the real world with a higher level of "black boxness," i.e., models in which even access to contextual embeddedness is prohibited. Addressing such a case is a subject for future research.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The code used for the experiments in this paper can be accessed here : https: //github.com/yusukekwn/Black-Box-Forgetting.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 We proposed Black-Box Forgetting, a novel problem of selective forgetting for black-box models. We introduced Latent Context Sharing (LCS), an efficient and effective parametrization method of prompt, which is suitable for derivative-free optimization. Experimental results demonstrated that our method outperforms the reasonable baselines with significant margins. In addition, the sensitivity analyses showed the effectiveness of our LCS.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: See Sec. 4.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We report error bars of the experimental results.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: See Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: Our research comforms to the NeurIPS code of ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: See Sec A.1.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We respect them properly.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: A.1 Broader ImpactsIn this paper, we introduced the novel problem called Black-Box Forgetting, i.e., the task of selective forgetting for large-scale and black-box pre-trained models, and demonstrated the potential effectiveness of parametrizing the latent contexts in the text prompts into unique and shared components. We here would like to emphasize several potential benefits of our work.1. Toward addressing the "Right to be Forgotten": It should comply with the request that if a service provider is asked to remove information so that their model cannot recognize certain information. This can be accomplished by training the model from scratch by removing samples of that class from the training data. However, retraining a large-scale model consumes an enormous amount of energy, which should be avoided. Selective forgetting may provide an efficient solution to this problem. 2. Toward efficient large-scale pre-trained models: Improving the efficiency of large-scale pretrained models is of current interest to many researchers. Various attempts have been made such as model compression and architecture optimization (e.g., https://sites.google. com/view/elvm/program). Meanwhile, as the "scaling law" indicates, the reasonable size of a model correlates with the amount of knowledge it memorizes. Therefore, if the number of classes (amount of knowledge) that can be recognized by the model is limited, the model can be scaled down accordingly, thereby improving the efficiency of the model. This contributes to expanding the applicability of large-scale pre-trained models. 3. Toward better control over text-to-image generation: While diffusion-based text-to-image generation can generate diverse types of high-quality images, controlling the content of images remains a challenge. Recent research has focused on "forgetting" visual concepts in order to avoid generating undesirable content [Graves et al., 2021 , Lu et al., 2024 , Zhang et al., 2024] . These methods forget certain classes by directly fine-tuning the diffusion model, but tuning the diffusion model itself is often costly. In contrast, given that typical generative models use a text encoder of a pre-trained VLM (e.g., CLIP) for conditioning, our method may provide an efficient approach to class forgetting by fine-tuning only the prompts of the text encoder.We believe that our paper will open new directions for these important problems of interest to the machine learning community, even if these are not immediately feasible with this paper alone.We conduct experiments on ImageNet-1k [Deng et al., 2009] to verify the effectiveness of our method on a larger-scale dataset. The setup is the same as the cases for CIFAR-100 and CUB-200-2011 (see Sec. 4.1) . The results are shown in Table 3 . Our method outperforms all the baselines in terms of H and Err for , which supports the effectiveness of our method further. 1 and 3 shows that our method performs poorly in Acc mem than the baseline methods. This is because Err for and Acc mem are in a trade-off relationship, with Acc mem tending to decrease as Err for is increased. This is presumably because features between classes are not completely disentangled in the feature space, so forgetting one class may negatively affect other classes (just as dog and cat share some common features). To provide justification for this, we report the results of using a loss more prioritized (weighted) for Acc mem in our method in Table 4 as "Ours (Acc prio.)." We can see that Ours (Acc prio.) outperforms all the other methods in Acc mem , with sacrificing Err for . Notably, both Ours and Ours (Acc prio.) outperform BBT and CBBT in H, indicating that our method achieves a better trade-off than BBT and CBBT. Furthermore, the results on CUB-200-2011 in Table 1 show that the white-box method CoOp [Zhou et al., 2022b] only improves Acc mem by 0.2% from zero-shot and our method even hurts Acc mem .Our problem is to achieve forgetting only specified classes while maintaining the accuracy of the other classes to be memorized, i.e., to improve Err for while maintaining Acc mem . Achieving both of these is more challenging than merely improving Acc mem alone. To verify our argument here, we report an analysis in Table 5 . "CoOp (White-Box w/ only memorization)" shows the results when white-box tuning by CoOp is performed to minimize the cross-entropy loss over only the classes to be memorized, i.e., forgetting is not performed. We can see a significant improvement in Acc mem . This result proves that, even in a white-box setting, achieving both improving Acc mem and Err for is more difficult than merely improving Acc mem alone. Since the black-box setting is generally more challenging than the white-box setting, it is not at all surprising that our method leads to a slight degradation in Acc mem . In this paper, we consider a few-shot learning scenario, i.e., we can access a small number of training examples for tuning the latent contexts. However, in some practical cases, one might be faced with a zero-shot scenario, i.e., a situation where no sample is available. We here consider a simple zero-shot tuning approach to the black-box forgetting task by only using the class names (i.e., class embeddings). Specifically, let z c and z denote the class embeddings before and after prompt tuning for the class to be forgotten, respectively. We assume only z is trainable and aim to tune z by minimizing the following negative NT-Xent loss:EQUATION)where z i is the class embedding for the i-th class to be kept memorized. This loss requires z to be orthogonal to z c as well as be similar to the embeddings of the other classes {z i }. By minimizing the loss, we can expect that the embedding of the class to be forgotten z c is placed equidistant from the embeddings of all the other classes {z i }.The results are shown in Table 6 . We found that the zero-shot approach (C-Emb.) performs significantly poorly compared to our few-shot learning approach. A closer look at each evaluation metric shows a slight advantage of C-Emb. over Ours in Err for , but C-Emb. is far lower than Ours in Acc mem . This is probably due to the nature of the negative NT-Xent loss, whereby the embedding of the class to be forgotten is closer to the embeddings of the other classes to be memorized, resulting in more misclassifications. These results prove that tuning with only the class embeddings does not provide satisfactory performance. The above results give us further inspiration; If training samples are available for some classes and not for the rest, would there be a benefit from combining the two approaches? We experimented under conditions in which half of the classes to be forgotten had training samples available and the rest did not. We use Ours for the classes for which the training samples are available and for the classes with no training samples. The result on ImageNet30 are shown in Table 7 . While Ours performed forgetting for only the classes with the training samples based on the loss given in Sec. 3.2, Ours + C-Emb. performed forgetting for all the classes by incorporating Eq. 3. We see that Ours + C-Emb. could outperform Ours in all the metrics, which proves the effectiveness of the combination. A.5 Forgetting vs. Learning from ScratchAn obvious alternative to forgetting that produces a model that can recognize only the classes to be memorized while preventing the recognition of only the classes to be forgotten is to learn the model from scratch for only the classes to be memorized. To clarify the benefit of forgetting, we evaluate the accuracy of ResNet-18 and ViT-B/16 trained from scratch over only the classes to be memorized. The results are shown in Table 8 . As can be seen, both models cause severe overfitting to the training data. That is, while these models achieve reasonable accuracy on the training data, they exhibit severely poor performance on the test data. We also tested both models when initialized with ImageNet pre-trained weights. While the results are improved to some extent for ResNet-18, these are still far behind our forgetting method. The reason for this is that, following the common protocol in context optimization [Zhou et al., 2022b] , we conducted our experiments in few-shot scenarios as explained in Sec. 4.1, which overwhelmingly lacks the number of training samples to learn the weights for even ResNet-18 and Vit-B/16. These results demonstrate the superiority of our forgetting approach, which achieves effective tuning with a small sample size.Another alternative approach to forgetting would be to only fine-tuning the model over the classes to be memorized. That is, do nothing to the classes to be forgotten and cause catastrophic forgetting. We test the performance of fine-tuned CLIP over only the classes to be memorized (Fine-tune CLIP).The results are shown in Table 9 . Compared to our method, which explicitly facilitates forgetting of the classes to be forgotten, catastrophic forgetting alone (Fine-tune CLIP) is not sufficient to achieve satisfactory forgetting performance. Also, our method is comparable to the Fine-tune CLIP in Acc mem except for CUB-200-2011, which does not significantly impair the classification accuracy for the classes to be memorized. These results support the validity of our forgetting approach. We evaluate another white-box forgetting approach that learns latent contexts through CoOp while also updating the model parameters. The results are shown in Table 10 . We see that simultaneously updating the model parameters does not improve performance, but rather hurts it. This is not surprising, as it is known that straightforward fine-tuning of the zero-shot CLIP does not improve performance [Wortsman et al., 2022] . As explained in Sec 3.1, each of the (d u + d s )-dimensional latent context is projected into the original D-dimensional context by a random projection A ∈ R D×(d u +d s ) sampled from a normal distribution N (0, σ), where σ is the standard deviation of the context embeddings. We analyze the impact of the choice of σ on the final performance. Table 11 shows the results on CIFAR-10 when σ is fixed to 1. We can see that the choice of σ has a significant impact on the performance in all the evaluation metrics. A.9 Performance for Different Choices of The Classes To Be ForgottenIn the experiments in the main paper (Sec. 4), 40% of the classes in each dataset were selected as the classes to be forgotten, in ascending order of the class index, starting with the 0-th class. We here investigate the performance of our method when using different selections of the classes to be forgotten. The results are in Table 12 . We see that our method achieves satisfactory selective forgetting performance regardless of the choice of the classes to be forgotten. {1, 2, 3, 4, 5, 6, 7, 8} 97.50 ±0.00 97.69 ±0.02 97.32 ±0.02 {2, 3, 4, 5, 6, 7, 8, 9} 97.94 ±0.01 97.76 ±0.01 98.12 ±0.02 {1, 2, 3, 4, 5, 6, 7, 8, 9} 99.51 ±0.00 99.09 ±0.01 99.93 ±0.00 {0, 2, 3, 4, 5, 6, 7, 8, 9} 98.83 ±0.02 98.55 ±0.02 99.10 ±0.01We use a single GV100 GPU with 12.885GB memory for all the experiments. Approximate computation time is 120 minutes for CIFAR-10, 400 minutes for ImageNet30 and 600 minutes for CIFAR-100 and CUB-200-2011.Justification: We do not include theoretical results. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.Note that the problem focused on in this paper is closely related to but different from the typical "machine unlearning" problem, which is the task of removing an arbitrary sample from a pre-trained model, i.e., obtaining a model that is identical to the model trained from scratch without that sample[Cao and Yang,2015, Golatkar et al., 2020a, Sekhari et al., 2021, Bourtoule et al., 2021, Golatkar et al., 2021, Kurmanji et al., 2023].This assumption is reasonable because the experiments in Sec. 4.3.1, 4.3.2 show its effectiveness.The actual amount of calculation is O(d 2 ) as CMA-ES estimates the covariance matrix internally.4, 5 https://www.cs.toronto.edu/~kriz/cifar.html 


Decision: This paper explores the selective forgetting problem for large-scale pre-trained models. In the paper, the authors introduce a novel method based on the derivative-free optimization of a learnable text prompt. Initially, this paper received mixed ratings. During the discussion period, the authors and the reviewers had in-depth discussions, and all the reviewers agreed that this paper is good enough to be presented at NeurIPS. The authors are required to include the discussions with all the reviewers in the final version.

Reviews: review: paper_summary: paper_summary: The authors propose to apply selective forgetting to black-box pretrained models, instead of the usual white-box settings. Since it is a black-box method, there is no parameter update and instead the prompts are the ones being optimized to decrease the performance on the target class to be forgotten. This is done by using an existing covariance matrix adaptation, which is a derivative-free optimizer. The paper describes the issues of optimizing a large dimensional space while gradients are not available, and the problem of having samples from the target forgotten class being mostly pushed towards a close-by, non-forgotten class instead. The first issue is solved by splitting the latent context between unique and shared, and reparametrizing them individually to lower dimensions. The second issue is solved by excluding the label information from a classic CE loss, and adding it as regularization for maximizing the entropy of the overall confidence.
strength_weakness: strength_weakness: Despite the main idea being clear and most of the method being well described, the writing could use some revision. CMA-ES is quite a key component of the proposed strategy, but is covered quite shortly. The LCS (ii) part is a bit over-complicated for such an easy concept. I think it could be explained a bit more elegantly and would highlight the usefulness and easiness of having this type of parametrization. Same goes for some of the subsections within the Analysis (4.3). I assume that the lack of space was the reason why some of the concepts from the compared methods are just mentioned and not explained in detail. However, in order to make it easier for the reader to understand what the analysis refers to and drive some of the conclusions, it might be better to just move one or two of them to the appendix. My suggestion would be moving 4.3.1 and 4.3.5 to the appendix, and using that space to improve the analysis of the others.

Nomenclature in some cases is a bit confusing. In figure 1, the losses are called forget and memorize, while in equations (1) and (2*missing) they are called uniform and CE. Same with z having subscripts or superscripts depending on BBT or LCS.

Text in figures 3 and 4 is too small.
questions: questions: 1. In the introduction, it is shortly mentioned, but I would like some more context on how close are some of the machine unlearning settings to the one proposed in this submission.

2. Footnote 2 is a bit weak. How are the results shown in section 4 enough reason to assume that there is a composition of ULC and SLC?

3. Why are the latent contexts optimized for the hyperparameters used? Are those decided under a validation set or overfitted to the test? Same for the number of latent contexts.

4. In appendix A.2, what is the reasoning for the groupings of classes to be forgotten? Just random? Or do they relate in some way that can support the use of equation (2*missing) versus (1)?
limitations: limitations: Regarding the experimental statistical significance, the authors mention that they do not report error bars, and I would also consider that such set of experimental settings would require the need to run each method multiple times, specially when there are some results that are quite comparable and could lead to statistical significance showing that some methods is not significantly different than another under some metrics.

This is such a strong point, in my opinion, that it is the reason why I rate the paper down from a 7 (accept) to a 6 (weak accept). Given that the paper relies quite heavily on its experimental analysis to showcase the interesting proposed scenario compared to current zero-shot or white-box approaches. Also in the appendix, there is this random sampling, which does not report how many times it is sampled either. Finlly, the point from question 2 is also important to showcase the limitation on the argumentation of the existence of ULC and SLC.
paper_summary: paper_summary: This paper addresses the problem of selective forgetting of specified classes, which involves tuning a pre-trained model to reduce the classification accuracy for only the specified classes without affecting the accuracy for the others. The proposed model introduces a novel method for Black-Box Forgetting based on the derivative-free optimization of a learnable text prompt. It introduces Latent Context Sharing (LCS), a novel parameterization method for contexts, which mitigates the difficulty of high-dimensional optimization using derivative-free optimization.
strength_weakness: strength_weakness: 1- The proposed approach for forgetting selective classes from the pre-trained model requires data samples for each elective class. Since it is applied in CLIP-based multi-modals, where the text encoder is also available, one might wonder why data samples are necessary for each forgetting class. Could the same result not be achieved using just the class name?

2- The proposed approach claims that it introduces Latent Context Sharing (LCS), a novel parameterization method for contexts, to mitigate the difficulty of high-dimensional optimization. However, considering that the embedding dimension in CLIP-based models is 512, which is not excessively large, and that this approach reduces the dimension to 100 for the CUB dataset, the reduction does not seem significantly lower than the original CLIP dimension. Therefore, in the case of larger datasets, this reduction might not have a substantial impact compared to the original CLIP dimension.

3- The proposed approach claims to be a black box due to the unavailability of pre-trained model information, such as architecture, parameters, and gradients, during training. However, my concern is that if you are given the pretrained model, extracting information about the model architecture and its parameters can still be quite challenging?

4- How is the proposed black-box forgetting different from machine unlearning?

5- I recommend conducting experiments on more diverse datasets, such as ImageNet-1K, to gain a better understanding of this approach.

6- Why is Acc_mem lower than the baseline approaches for each dataset?

7- Can the proposed approach be applied in a continual learning setup?
questions: questions: Please address all my concerns raised in the weaknesses section.
limitations: limitations: My concerns raised in the weaknesses section described the limitations of the proposed approach.
paper_summary: paper_summary: This paper studies the black-box forgetting problem. To this end, the authors optimize the input prompts with a proposed latent context sharing scheme by CMA-ES optimization. The authors achieve the selective forgetting goal by minimizing the cross-entropy loss on memorized classes and maximizing entropy loss on forgetting classes. Experiments demonstrate the promising of the proposed method.
strength_weakness: strength_weakness: The problem setting of this work does not convince me, possibly due to the presentation of the paper. I hope the authors can provide more justification regarding this.

Experiments could be further enhanced, as my detailed comments below.
questions: questions: Q1.

*“—Retaining the classes that do not need to be recognized may decrease overall classification accuracy, as well as cause operational disadvantages such as the waste of computational resources and the risk of information leakage”* 

How do you define the overall accuracy? Is it the overall accuracy of all classes of the pre-trained model, or the overall accuracy of downstream tasks, such as CIFAR-10 in the experiments? This is quite confusing for me. If it is the latter, during the fine-tuning process on downstream tasks, it seems that tuning only the necessary classes (those to be recognized) can achieve the best performance. Why must we forget some classes?

Moreover, the phrase ‘waste of computational resources’ is also confusing. Learning on all classes is conducted during the pre-training phase and cannot be modified by the proposed method. Then, during fine-tuning, why would forgetting to reduce computational resources? **Aside from preventing information leakage**, what is the rationale behind forgetting certain classes?  i.e., When we fine-tune a CLIP model on CIFAR-10, what is the real and sound motivation for us to forget some classes of CIFAR-10?

Q2. If the final goal is to forget 40% of classes and memorize 60% of classes on CIFAR-10, why must we fine-tune CLIP? Why not directly train a new model (with the same architecture as CLIP or a smaller ResNet or ViT) with only the 60% memorized classes? It would be better to include the results of this baseline in Table 1. I assume that the authors believe a model fine-tuned from CLIP could achieve better performance on the memorized classes than a model trained from scratch. If this is the case, experiments on CIFAR datasets alone are insufficient to demonstrate this, as these data are relatively easy. Experiments on more complex datasets/tasks, such as those mentioned in autonomous driving, are necessary to further help to justify the superiority of the proposed method.

Q3. Can the proposed method be extended to more complex tasks such as segmentation or object detection? As the authors mentioned, in an autonomous driving system, it is sufficient to recognize a limited number of classes such as cars, pedestrians, and traffic signs.  In autonomous driving, the task is often more complex than pure classification, such as 2D and 3D object detection tasks.

Q4. How about the performance of Ours vs. Solely Fine-tune CLIP on memorized classes? It would be much better if this result is verified on more complex datasets.

Q5. In the white-box setting, one can update more model parameters beyond just the input prompts. How would the performance change if we update all model parameters for CoOp (White-Box)?


I would like to increase my score if the authors could convince me regarding the problem setting.
limitations: limitations: The proposed method is not fully black-box, as it still requires access to the embeddings and output logits, which are unavailable in a fully black-box setting, similar to the GPT-4 API. However, I understand that achieving a fully black-box setting is very challenging, and I do not consider this a reason for rejection. I mention this point merely as a slight limitation.
paper_summary: paper_summary: This paper explores a selective memorization problem of classification models under a black-box setup. The proposed method selectively applies different learning objectives, cross-entropy minimization, and entropy maximization for classes to be memorized and classes to be unmemorized, respectively; they become guidance for a small amount of text embedding tokens (prompt) by a black-box optimization algorithm, CMA-ES. The authors evaluate the proposed method in terms of selective memorization capability.
strength_weakness: strength_weakness: # 1. Lack of novelty
* The proposed method can be viewed as a variant of BBT [1], which explicitly separates the seed latent context into shared parts and unique parts. Although this is a new application of BBT in selective memorization scenarios, if the authors want to claim the novelty of the modified BBT method, I think they should further provide analyses on efficiency and qualitative results on learned context to contrast with BBT.
* Moreover, the proposed learning objective -a combination of cross-entropy minimization and entropy maximization- is also quite common [2], so it is hard to claim its originality from this work.

# 2. Inappropriate validation setup
* As the main goal of this study is selective memorization of classification models, I expected existing methods for selective memorization and unlearning to be the natural baselines. However, all the comparisons are conducted by training the black-box tuning model via the combination of cross-entropy and entropy. Comparing the proposed method with other available unlearning objectives would be necessary to measure its validity.
* The scope of validation is also limited to three natural object recognition. It is important to validate the methods across a more diverse range of image domains to robustify the conclusion.

---
> Reference
* [1] Black-Box Tuning for Language-Model-as-a-Service, Sun et al. 2022
* [2] Mitigating Information Leakage in Image Representations: A Maximum Entropy Approach, Roy and Boddeti 2019
questions: questions: - Is the sign of $L_{uniform}$ term in the final objective a typo? To maximize entropy, I think we should minimize $-L_{uniform}$ to maximize entropy.
- For selective memorization, I believe that not only the unmemorization for the selected classes but also the memorization for the remaining classes is crucial. However, in Table 1 CUB-200-2011 results indicate that the white-box method only improves $Acc_{mem}$ 0.2 % from zero-shot, which is a very trivial amount given that the strong few-shot learning capability of CoOp [3] and CoCoOp [4] on the similarly fine-grain visual recognition tasks.  Furthermore, the proposed method even hurts the $Acc_{mem}$ from the zero-shot model. Could you explain why the proposed learning objective shows poor learning capability on the CUB dataset compared to the CIFAR datasets?

---
> Reference
* [3] Learning to Prompt for Vision-Language Models, Zhou et al. 2022
* [4] Conditional Prompt Learning for Vision-Language Models, Zhou et al. 2022
limitations: limitations: The authors adequately stated the limitation in the last section.

Comments: title: Response to Reviewer SKod's comment
Additional responses.
Our responses to your additional questions (a) and (d).
Our responses to your additional questions (b) and (c).
Thank you, and our response to your additional question on Table E.
Response to authors
Further response to the question of the novelty of the Black-Box Forgetting problem
comment: Dear ACs and Reviewers,

Reviewer-Author Discussion is almost over. Once again, we would like to thank ACs and Reviewers for devoting a great deal of time and effort to our paper. We have received many constructive comments from the reviewers through this discussion period, which significantly improves the final version of our paper. We believe that we were able to thoroughly address all of the questions posed by the reviewers with the best effort we could in this short rebuttal period. 

Furthermore, as some of the discussions we have had here have, our paper has a potential to bring new perspectives to the machine learning community even beyond the scope of our paper, and serve as sources of inspiration for many researchers.

Many Thanks,

Authors
We would like to thank Reviewer SKod for your patience and your consideration during this rebuttal period. But could you please allow us to provide one last response, because we feel that the reviewers' view is somewhat biased, saying like "good methods should not be inspired by empirical observations." 


Looking at the facts of the past, there are a number of good methods built on empirical observations (and are not designed specific to the target tasks). For example, Focal Loss was originally proposed for object detection, but it is actually a loss to mitigate class imbalance, and its connection to object detection is rather weak [a]. ArcFace [b] was proposed for face verification, but its core is a variant of max margin loss, which is not really relevant to faces. 


Our method is designed based on sharp observations in forgetting, which in itself should not be punished. Furthermore, even if the reviewer could not agree that the LCS is designed for forgetting, at least could agree that it is designed for the Black-Box setting, which is the other half of the focus of this paper.


[a] Lin et al., Focal loss for dense object detection, in Proc. ICCV, 2017.

[b] Deng et al., ArcFace: Additive Angular Margin Loss for Deep Face Recognition, in Proc. CVPR, 2019
I am so sorry for my decision-invariance attitude, but I want to adhere to my rating.

These long discussions might cause your mind severe damage, and I am regretful for keeping my statement due to the unsolved concern.

I can not agree with the authors' claim that "LCS is designed for forgetting." Figure 5 is just an empirical result, where the authors provide post-experiment interpretation based on the results.

To be specific, it is only a consequential interpretation, and the design of the methodology cannot be justified as it is for forgetting. That is why I think the connection between the problem definition and the derived methodology is weak, and I will try to stick to this position. 

However, since I found some good implications from this study and other reviewers are looking positively, I will not be upset even if this paper is accepted.

Sorry again,

Reviewer SKod
Thank you for your response. We are very happy that now Reviewer SKod acknowledged the novelty of the Black-Box Forgetting problem.

Let us clear up the reviewer's misunderstanding: LCS is designed for forgetting. Please see Figure 5 in our original paper. The results show that as the ratio of the number of classes to be forgotten $r_\text{for}$ increases, the forgetting performance $Err_\text{for}$ of the existing parametrization method, BBT, tends to decrease. The only way to improve the $Err_\text{for}$ of BBT is to increase the number of learnable parameters $m$ (as can be seen in Figure 3), which clearly suggests the limitation of the scalability of BBT. Based on this observation, we developed our LCS to fundamentally improve the scalability of parameter representation for improving $Err_\text{for}$, that is, for forgetting.

Note that the reviewer's comment is never a disadvantage of our LCS, even though it may suggest that LCS can be applied to other problems, besides forgetting.
The authors' statement on the novelty of problem setup is somewhat convincing, and I want to express my huge gratitude for the extensive responses. However, I would regretfully say that I want to keep my score.

This black-box forgetting problem has not been explored yet, and the authors address it with their new method.
The proposed method, latent context sharing (LCS), is designed to reduce the number of learnable parameters for better black-box optimization.
I still do not see the necessity of this proposal, and there is a missing link in the connectivity between LCS and the black-box forgetting problem.

To be specific, the LCS method is not tailored for "forgetting," and it can be regarded as a method for improving parameter efficiency that will be applied to other PEFT scenarios.
The weak connection between the LCS method and the problem setup is the main reason that I kept my score towards rejection.
Thank you again for your feedback. We are pleased to hear that we were able to properly address your questions and that you are in favor of accepting our paper.

**To (c)**: Thank you for drawing our attention to this interesting point. We will try to put more detailed discussions in the final version, but our preliminary idea in the case of pre-trained VLMs at present is as follows. 

Contrastive language-image pretraining learns to align image and text features in a common space using a massive dataset. Thus, the common and unique components of the text features are expected to be aligned with the image features as well, even if the modalities are different. Empirically, image-specific "levels" seem to be aligned with the text features, at least partially, but surprisingly very well. For example, it has been shown that the pre-trained CLIP can recognize an object in a local region of an image, suggesting that the text features are learned to absorb differences in scale [a]. CLIP guidance can produce an image that is faithful to the prompt by moving the image feature in the direction of the prompt embedding [b]. These results may support the earlier expectation and may also be a reason that effective forgetting is possible for pre-trained VLMs.

**To (a) and (d)**: We will make sure to add the results on ImageNet-1K in the final version. Thank you.

[a] Miyai et al., LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learning, in Proc. NeurIPS, 2023.

[b] Nichol et al., GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models, in Proc. ICML, 2022.
__following on (a) and (d)__: yes, that is what I meant. I also agree that it is a larger question than what the paper covers currently. I appreciate the results provided and the thorough explanation with the new results. I agree that the insights from the discussion are indeed of interest for further research!

Overall, my questions have been answered. Thanks to the authors for their thorough responses. Some clarifications within the discussion would be needed for the final version, but I have no further questions. I think the questions from the other reviewers and the author responses to them are also satisfactory. Some of them highlight some of the limitations of the proposed setting and also should be included in a final version. The only comment from a reviewer that I also consider important to note, is how simple the datasets used are (Q4 from SKod), and agree with the observation. The results on ImageNet-30 by the authors are good step to solve that issue, although I would be cautious of results obtained during a rebuttal (not due to suspicion of malice, but just due to the fast implementation). However, I am satisfied with the experiments provided. Therefore, I would be in favour of accepting the paper.
__following on (c)__: I get the idea for word embeddings, or even when working with multiple modalities, where there are shared spaces across a modality, and specific features of each modality separately (like audio having tone specific features unused by text). However, within images, it is not so intuitive on how those spaces get disentangled between the shared and specific/unique based solely on the black-box output of classification layers, which represent orthogonal dimensions (due to the CE-loss). Specially if we think about how the black-box model does not provide information of the levels[^1] at which features are agreggated. This should have a direct effect on how knowledge relates within the model, and how it (or its paths) can be forgotten. It seems to require some knowledge about the output space characteristics of the black-box. How the knowledge is learned, like if the black-box is a mixture of experts, network in network, or other unknown settings, might also have an effect to how the shared or unique contexts have an effect.

[^1]: here I mean field of view, scale of the pattern, or semantic level of the involved features.
**To (a) and (d):** We appreciate you sharing these interesting questions with us.

We assume that your intention behind these two questions is that "the proposed method performs prompt tuning with a few training samples, but to truly realizing class-selective forgetting, it would be necessary to control the class distribution, rather than relying on training samples." Given that the fundamental goal of machine learning is to obtain generalization performance from a limited number of training samples, it would be rather difficult to cover this question in this paper alone, but we will try to discuss as much as possible.

First of all, would it be possible to tune a pre-trained VLM without using training samples? As is well known, one advantage of tuning the prompt rather than the model parameters is that generalization performance can successfully be improved even when only a few training samples are available [a]. However, prompt tuning without using training samples, i.e., "zero-shot prompt tuning," is still an open problem; effective tuning is difficult unless some external knowledge is available, e.g., LLMs [b] or style-specific prompts [c].

In fact, we tried to answer Q1 of VErt by updating the class embeddings by minimizing the negative NT-Xent loss, without using training samples. Indeed, the reason for using the negative NT-Xent loss was exactly what you intended, i.e., we aimed at controlling the distribution of the class embeddings without relying on any training samples. More specifically, to ensure that the embedding of the class to be forgotten is placed equidistant from the embeddings of all the other classes. We believe this is a reasonable idea, provided that no external knowledge is available.

However, as the results show in Table B, $Acc_\text{mem}$ was much lower than when using the training samples. This is probably due to the nature of the negative NT-Xent loss, whereby the embedding of the class to be forgotten is closer to the embeddings of the other classes to be memorized, resulting in more misclassifications.

Given these observations, it would be challenging to fully control the class distributions without any training samples. This also implies that developing new evaluation metrics that do not rely on test samples is rather challenging for now, although it is an interesting problem. However, since classification performance is usually evaluated on test samples, we believe that the evaluation metrics we used in our paper are at least reasonable.

Next, how should we decide whether to use given training samples or to directly control the class embeddings? The advantage of controlling class embeddings is that forgetting can be achieved even for the classes without training samples. Therefore, a natural recommendation comes up in our mind is to use training samples for the classes for which those are available, and to directly control the class embeddings for the classes with no training samples.

To test the validity of this recommendation, we experimented under conditions in which half of the classes to be forgetting had training samples available and the rest did not. The results on ImageNet30 are shown below.

| Method                   |    $H\uparrow$    | $Err_{\mathrm{for}}\uparrow$ | $Acc_{\mathrm{mem}}\uparrow$ | 
| :----------------------- | :-------------------------: | :----------------------------: | :----------------------------: | 
| Ours + C-Emb.            | $92.30_{\pm 0.05}$ |      $86.56_{\pm 0.08}$      |      $98.87_{\pm 0.00}$      | 
| Ours                     | $89.34_{\pm 0.03}$ |      $81.56_{\pm 0.05}$      |      $98.78_{\pm 0.00}$      |

While Ours performed forgetting for only the classes with the training samples based on the loss given in Sec. 3.2, Ours + C-Emb. performed forgetting for all the classes by incorporating the negative NT-Xent loss. We see that Ours + C-Emb. could outperform Ours in all the metrics, which proves the effectiveness of our recommendation. 

We would like to thank you for your question which led us to discover a promising extension of our approach, Ours + C-Emb. which was not discussed in our original paper! While the development of effective zero-shot prompt tuning is beyond the scope of this paper, we believe it is healthy that such a valuable discussion will arise in the community as a result of our paper, which is our pleasure. So we would like to add the above discussions in the final version.

[a] Zhou et al., Learning to Prompt for Vision-Language Models. IJCV, Vol. 130, No. 9, pp. 2337–2348, 2022.

[b] Menon and Vondrick, Visual Classification via Description form Large Language Models, in Proc. ICLR, 2023.

[c] Cho et al., PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalization, in Proc. ICCV, 2023.
Thank you for your detailed review not only of our responses, but also of the other reviewers' comments and our responses to them. We are very pleased to hear that most of our responses were satisfactory. Below we would like to respond to your additional questions, first to (b) and (c), followed by (a) and (d).

**To (b)**: Yes, we will make sure to add the results and discussions, as well as the detailed evaluation protocol in the final version.

**To (c)**: Sorry for the lack of a detailed explanation. The following statement, which is originally provided as our response to question 2, is the inspiration behind our design. Please let us know if it is still not convincing to you.

> The key idea is to assume shared parameters among different latent contexts. This inspiration comes from successful word embedding methods; most word embedding methods are trained on the assumption that locally co-occurring words have semantic correlations between them (e.g., [a-c]). This inspires the idea of explicitly modeling semantic correlations between words in a prompt as shared components.

[a] Mikolov et al., Efficient Estimation of Word Representations in Vector Space, arXiv pre-print 1301.3781, 2013.

[b] Pennington et al., GloVe: Global Vectors for Word Representation, in Proc. EMNLP, 2014.

[c] Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, in Proc. NAACL-HLT, 2019.
Thanks for the author's response. I agree that answers to Q1-Q5 will help improve the manuscript. Specialy, the context on machine unlearning, and the differences from the proposed black-box forgetting scenario are relevant, and the proposed changes on literature review are well covered. However, reading through the other reviewer's and response, I would stress the need to highlight well those differences.

(a) As further discussion, when saying "_Machine unlearning aims to remove the influence of specified training samples on the training model, whereas Black-Box Forgetting aims to prevent the recognition of specified classes._", it raises the question of how that affects the evaluation. If the shift between the two problems is going from specific samples to whole classes, how well do the metrics proposed evaluate the new scenario? The metrics used on the paper are based on the accuracy over specific samples, instead of measuring the shift between using the learned latent contexts and not. I understand that the metrics on single samples are aggregated, I am referring towards having a better metric to measure that the distribution of the specified class is forgotten.

(b) Q6, Q7 and Q8 are satisfactory responses for the experimental setting and evaluation. I would support adding those details to the final version, since the scenario should be clearly stated for further reproducibility. I appreciate that the results are over more than one seed, and I think the low standard deviation is a positive sign for the conclusions of the experimental section and reinforce the conclusions.

(c) My original question 2 is unaddressed. The paper states "_We assume that each latent context is composed of unique components (Unique Latent Contexts (ULC)) and common components (Shared Latent Context (SLC)) among multiple latent contexts_" with a footnote saying that the assumption is true due to the experiments in Sec. 4. However, this seems like a very weak argument. How did this assumption came to be, or what supports it, even before having an experimental support for it?

(d) Reading through some reviews and rebuttals, the author's have also clarified some of the interesting questions raised by fellow reviewers, which in most cases are satisfying. Answer to Q1 of reviewer VErt is not very convincing. If the proposed scenario distinguishes itself from machine unlearning by moving away from specific samples, the idea of using the prompts becomes much more interesting, instead of keeping samples. The authors address on their response the performance improvements, but the interesting part of the question is not the numbers, but challenging the decision of using the given samples vs. sampling the prompts for the classes involved.
We would like to thank you for your great effort in carefully reading our rebuttal and for your sincere consideration of it. We are very happy that our response has successfully resolved your questions. Of course, we will make sure to include our new experimental results, discussions, and explanations we provided in our rebuttal in the final version.

Regarding your additional question about the results in Table E, our response to **Q6 of Reviewer VErt** should be highly relevant. That is, Table C in the attached PDF shows that $Err_\text{for}$ and $Acc_\text{mem}$ are in a trade-off relationship, i.e., Ours ($Acc$ prio.), which is optimized by prioritizing $Acc_\text{mem}$, successfully improves $Acc_\text{mem}$, but at the expense of $Err_\text{for}$. This suggests that the underlying reason why the pre-trained model in Table E achieved higher $Acc_\text{mem}$ than Ours is probably because Ours was forced to sacrifice $Acc_\text{mem}$ to increase $Err_\text{for}$. 
Nevertheless, our method achieves a better trade-off (i.e., higher $H$) than the existing methods (BBT and CBBT) in all of the experiments. Furthermore, despite being a black-box method, Ours ($Acc$ Prio.) achieves $Acc_\text{mem}$ comparable to CoOp, a white-box method, which emphasizes the effectiveness of our method as a black-box optimization method. We will add a clear discussion about this point in the final version as well. Thank you for your valuable suggestion again!
Thanks for the authors’ thoughtful response. With newly provided results, the paper becomes much stronger on the empirical side. The clarifications regarding the significance of Selective Forgetting on Generative Models make the problem setting more practical. Most of my initial concerns have been addressed. I shall increase my score and please include the new results and discussions/clarifications in the revised paper. 

Moreover, for results in Table E, when the pre-trained model’s performance is not good enough, solely fine-tuning on memorized classes still achieves much better Acc_{mem} than the proposed method, i.e., results on CUB-200-2011. This could be further discussed in the revision.
Thank you for taking the time to read through our rebuttal so promptly! We are very pleased to hear that most of your questions have been properly addressed.

Thank you also for clarifying your question about the novelty of our problem, Black-Box Forgetting. However, we would like to argue that novelty of a problem should not necessarily be judged based on the difference in performance between existing and proposed methods. In fact, looking back at the history of machine learning, even if the difference in accuracy between existing and proposed solutions is less than 5%, there are many studies that have brought about significant advances in community by introducing novel problems. For example, few-shot object detection [a], universal domain adaptation [b], and open-set semi-supervised learning [c], just to name a few (You would agree that it is not possible to determine a golden rule like "If and only if there is more than $x$% difference between the proposed and existing methods, the problem will be considered novel.").

We also want to argue that our Black-Box Forgetting is challenging. Besides Table 2 on CIFAR-10, below we report results for three other more challenging datasets.

| | |CIFAR-100| | |CUB-200-2011| | |ImageNet30| |
|:-------|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|
|Method|$H\uparrow$|$Err_{\mathrm{for}}\uparrow$|$Acc_{\mathrm{mem}}\uparrow$|$H\uparrow$|$Err_{\mathrm{for}}\uparrow$|$Acc_{\mathrm{mem}}\uparrow$|$H\uparrow$|$Err_{\mathrm{for}}\uparrow$|$Acc_{\mathrm{mem}}\uparrow$|
|BBT|$79.38_{\pm 0.01}$|$87.30_{\pm 0.01}$|$71.09_{\pm 0.00}$|$58.75_{\pm 0.01}$|$88.98_{\pm 0.04}$|$43.85_{\pm 0.01}$|$94.22_{\pm 0.05}$|$90.17_{\pm 0.08}$|$99.06_{\pm 0.01}$|
|BBT w/ Sep-CMA-ES|$69.29_{\pm 0.01}$|$67.83_{\pm 0.02}$|$70.81_{\pm 0.01}$|$53.74_{\pm 0.02}$|$74.72_{\pm 0.02}$|$41.96_{\pm 0.02}$|$91.18_{\pm 0.02}$|$84.44_{\pm 0.04}$|$\\textbf{99.07}_{\pm 0.00}$|
|BBT w/ VkD-CMA-ES|$75.41_{\pm 0.02}$|$79.56_{\pm 0.01}$|$\\textbf{71.67}_{\pm 0.01}$|$55.12_{0.02}$|$81.49_{\pm 0.03}$|$41.65_{\pm 0.02}$|$91.25_{\pm 0.05}$|$84.58_{\pm 0.09}$|$99.06_{\pm 0.00}$|
|Ours|$\\textbf{80.99}_{\pm 0.01}$|$\\textbf{93.37}_{\pm 0.02}$|$71.52_{\pm 0.01}$|$\\textbf{59.67}_{\pm 0.01}$|$\\textbf{89.29}_{\pm 0.01}$|$\\textbf{44.81}_{\pm 0.01}$|$\\textbf{97.28}_{\pm 0.01}$|$\\textbf{95.94}_{\pm 0.01}$|$98.67_{\pm 0.01}$|

As we can see, the variants of BBT do not provide stable performance over different datasets. In particular, the three BBT variants show $Err_\text{for}$ more than 5-10% lower than our method on ImageNet30. Furthermore, the results on CUB-200-2011 highlight the difficulty of the Black-Box Forgetting problem, as even our method yielding the best results could achieve only 59.67% in $H$. 

Moreover, as shown in Tables 1 and G, all the existing methods failed to achieve 80% in $Err_\text{for}$ for all the datasets. The performance degradation in $H$ is also significant, with $H$ values for BBT and CBBT being up to about 10% lower than for Ours. Figure 3 suggests that the scalability of BBT is poor, which is an inherent limitation that can be resolved by our method. 

Based on these facts, we believe that our Black-Box Forgetting is not an insignificant problem, but one that is both novel and worthy of technical consideration.

[a] Kang et al., Few-shot Object Detection via Feature Reweighting, in Proc. ICCV, 2019.

[b] You et al., Universal Domain Adaptation, in Proc. CVPR, 2019.

[c] Yu et al., Multi-Task Curriculum Framework for Open-Set Semi-Supervised Learning, in Proc. ECCV, 2020.
I appreciate the authors' comprehensive responses!
Most of my concerns are well-addressed.

About the novelty and contribution, I think if the authors want to claim the novelty of this work on the problem setup itself, i.e., Black-Box Forgetting, this problem should be challenging so that naive application of the existing method is struggled with.

For example, similar work on the Black-Box optimization setup [Oh et al. 2023] clarifies the problems of naive extension of the white-box methods [Bahng et al. 2022] and addresses those problems with their new method.
However, in Table 2. of the submitted draft, **BBT with a different configuration of CMA-ES, which is a naive extension of the existing method, already achieves very strong performance**. This raises doubt about the difficulty of the considered problem setup, and given that, I still lean towards the negative side because the proposed method, LCS, lacks the necessity.

Could the authors give further comment and opinion on this?

---

- Oh et al. 2023, BlackVIP: Black-Box Visual Prompting for Robust Transfer Learning
- Bahng et al. 2022, Exploring Visual Prompts for Adapting Large-Scale Models