SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Image Quality Metrics. First, we evaluate the image quality of the classes to remember using standard metrics such as the Fréchet Inception Distance (FID), Precision, and Recall [31, 32]. Ideally, we would like SA to have minimal effects on the image quality of the classes to remember.
Probability of Class to Forget. Second, using an external classifier, we evaluate generated samples from the class to forget to ensure that the class has been successfully erased. The probability of a class to forget is defined as  $\mathbb{E}_{p(\mathbf{x}|\theta ,\mathbf{c}_f)}[P_\phi (\mathbf{y} = \mathbf{c}_f|\mathbf{x})]$ , where the expectation is over samples generated from our trained model, and  $P_{\phi}(\mathbf{y}|\mathbf{x})$  is a pretrained classifier. If we choose  $q(\mathbf{x}|\mathbf{c}_f)$  to be an uninformative distribution, such as the uniform distribution, this should approach  $1 / N_{\text{classes}} (= 1 / 10$  for the datasets studied here) as the classifier becomes completely uncertain which class it belongs to.
Classifier Entropy. This is the average entropy of the classifier's output distribution given  $\mathbf{x}_f$ , defined as  $H(P_{\phi}(\mathbf{y}|\mathbf{x}_f)) = -\mathbb{E}_{p(\mathbf{x}|\theta ,\mathbf{c}_f)}[\sum_iP_{\phi}(\mathbf{y} = \mathbf{c}_i|\mathbf{x})\log P_{\phi}(\mathbf{y} = \mathbf{c}_i|\mathbf{x})]$ . When we choose  $q(\mathbf{x}|\mathbf{c}_f)$  to be the uniform distribution, all class information in the generated  $\mathbf{x}_f$  should be erased. The entropy should therefore approach the theoretical maximum given by  $-\sum_{i = 1}^{10}\frac{1}{10}\log \frac{1}{10} = 2.30$  as the classifier becomes maximally uncertain and assigns a probability of  $1 / 10$  to every outcome.
4.1 MNIST, CIFAR10 and STL10 Results
In this experiment on MNIST, CIFAR10 and STL10, we attempt to forget the digit '0' in MNIST and the 'airplane' class in CIFAR10 and STL10. We choose  $q(\mathbf{x}|\mathbf{c}_f)$  to be a uniform distribution over the pixel values. In brief, the results suggest that SA has successfully induced forgetting for the relevant class, with minor degradation in image diversity of the classes to remember. Qualitative samples are shown in Fig. 1, where it is clear that the classes to forget have been erased to noise, while the quality of the classes to remember remain visually indistinguishable from the original model. Additional samples are shown in Appendix D.1. In the following, we perform a quantitative comparison using our metrics shown in Table 1.
Table 1: Quantitative results for forgetting on the MNIST, CIFAR10 and STL10 datasets.  $H(P_{\phi}(\mathbf{y}|\mathbf{x}_f))$  and  $P_{\phi}(\mathbf{y} = \mathbf{c}_f|\mathbf{x}_f)$  indicate the entropy of the classifier's distribution and the probability of the forgotten class respectively. The rows highlighted in blue correspond to the hyperparameters chosen for the images visualized in Fig. 1. The rows highlighted in orange are ablation results for CIFAR10.
FID (↓)	Precision (↑)	Recall (↑)	H(Pφ(y|xf))	Pφ(y = cf|xf)
MNIST	Original	-	-	-	0.103	0.967
λ = 100	-	-	-	2.19	0.0580
CIFAR10	Original	9.67	0.390	0.788	0.0293	0.979
9 Classes	9.46	0.399	0.783	-	-
λ = 10	9.08	0.412	0.767	1.47	0.156
λ = 1	19.3	0.286	0.770	0.977	0.700
λ = 50	8.41	0.428	0.760	1.17	0.142
λ = 100	8.33	0.429	0.758	1.07	0.235
No GR (λ = 10)	126	0.0296	0.268	0.893	0.737
STL10	Original	14.5	0.356	0.796	0.0418	0.987
9 Classes	14.5	0.360	0.803	-	-
λ = 10	18.0	0.378	0.713	1.80	0.0189
First, we evaluate the information content left in the generated  $\mathbf{x}_f$  by examining the rows in Table 1 that are highlighted in blue. On MNIST, there is a  $96.7\%$  probability of classifying the samples of the '0' class from the original model correctly, and correspondingly a low entropy in its distribution. However, after training with  $\lambda = 100$ , the probability drops to  $5.8\%$ , while the entropy closely approaches the theoretical maximum of 2.30, indicating that any information in the generated  $\mathbf{x}_f$  about the digit '0' has been erased. We see a similar result for the CIFAR10 and STL10 diffusion models, where the entropy increases significantly after training, although it does not approach the maximum value as closely as the VAE.
Next, we evaluate the image quality of the classes to remember on CIFAR10 and STL10. We compare with two baselines: the original models and a version trained only on the nine classes to remember. Surprisingly on CIFAR10, training with  $\lambda = 10$  actually improves FID slightly, which a priori is unusual as the baselines should serve as natural upper-bounds on image quality. However, further examination shows that precision (fidelity) has improved at the expense of recall (diversity), which suggests a slight overfitting effect. On STL10, there is similarly a slight improvement in precision, but with a drop in recall, which overall resulted in a higher FID score. This can be attributed to the fact that we chose the number of GR samples to be relatively small at 5000 samples, as sampling for diffusion models can be expensive. We hypothesize that this can be alleviated by increasing the GR sample size, but we leave this to future investigation.
Ablations. We conduct ablations on the  $\lambda$  parameter and on the generative replay term in our objective function, using DDPM trained on CIFAR10. Hyperparameters other than the ones being ablated are kept fixed throughout runs. The results are highlighted in orange in Table 1. Starting with ablations on  $\lambda$ , we see that when  $\lambda = 1$ , image fidelity as measured by FID and precision is significantly poorer than at larger values of  $\lambda$ , showing that the FIM term is crucial in our training scheme. As  $\lambda$  is increased, there is a drastic improvement in fidelity, which comes at a slight cost to diversity as measured by recall, although the changes are relatively minor across the tested range of  $\lambda \in [10, 100]$ . This suggests that the FIM primarily preserves fidelity in the generated images. When comparing classifier entropy, we see that increments beyond  $\lambda = 10$  decreases entropy further from the upper-bound, which indicate some information leakage to the forgotten samples being generated. Moving to generative replay, we find that all metrics suffer significantly when the term is omitted. In summary, our ablation studies show that generative replay is crucial in our method, and intermediate values  $\lambda \in [10, 100]$  is sufficient for good performance.
4.2 Case Study: Stable Diffusion
"Forget Famous Persons. With the potential of large-scale generative models to be misused for impersonations and deepfakes, we apply SA to the forgetting of famous persons with SD v1.4. We leverage the fact that with language conditioning, we can choose  $q(\mathbf{x}|\mathbf{c}_f)$  to be represented by images that are appropriate substitutes of the concept to forget. For instance, we attempt to forget the celebrities Brad Pitt and Angelina Jolie, thus we set  $\mathbf{c}_f = \{\text{"Brad Pitt"}\}$  and  $\mathbf{c}_f = \{\text{"Angelina Jolie"}\}$ .
Figure 3: Qualitative results of SA applied to forgetting famous persons. Within each column, the leftmost image represents SD v1.4 samples, the middle image represents SA with  $q(\mathbf{x}|\mathbf{c}_f)$  set to "middle aged man/woman" and the rightmost image is SA with  $q(\mathbf{x}|\mathbf{c}_f)$  set to "male/female clown". [...] is substituted with either "Brad Pitt" or "Angelina Jolie".
Table 2: Quantitative results from the GIPHY Celebrity Detector. For SA, we use the variant with  $q(\mathbf{x}|\mathbf{c}_f)$  set to "middle aged man" or "middle aged woman" for forgetting Brad Pitt and Angelina Jolie respectively. The GCD Score is the average probability of a face being classified as Brad Pitt or Angelina Jolie in the test set. Numbers in brackets are standard deviations. Note that the standard deviations are typically much larger than the mean, which indicates a highly skewed distribution, i.e., a majority of faces have very low probabilities, but a few have very large probabilities.
Forget Brad Pitt	Forget Angelina Jolie
Proportion of images without faces (↓)	GCD Score (↓)	Proportion of images without faces (↓)	GCD Score (↓)
SD v1.4 (original)	0.104	0.606 (0.424)	0.117	0.738 (0.454)
SLD Medium	0.141	0.00474 (0.0354)	0.119	0.0329 (0.129)
ESD-x	0.347	0.0201 (0.109)	0.326	0.0335 (0.153)
SA (Ours)	0.058	0.0752 (0.193)	0.0440	0.0774 (0.213)
and represent  $q(\mathbf{x}|\mathbf{c}_f)$  with images generated from SD v1.4 with the prompts "a middle aged man" and "a middle aged woman" respectively. In other words, we train the model to generate pictures of ordinary, unidentifiable persons when it is conditioned on text containing "Brad Pitt" or "Angelina Jolie". In this way, our model still generates semantically-relevant pictures of humans, as opposed to uniform noise if we had chosen the same  $q(\mathbf{x}|\mathbf{c}_f)$  as the previous section.
To demonstrate the control and versatility of SA, we conduct a second set of experiments where we map the celebrities to clowns, by setting  $q(\mathbf{x}|\mathbf{c}_f)$  to images of "male clown" or "female clown" generated by SD v1.4<sup>1</sup>. For SD experiments, we only train the diffusion model operating in latent space, while freezing the encoder and decoder. Our qualitative results are shown in Fig. 3, where we see that the results generalize well to a variety of prompts, generating realistic images of regular people and clowns in various settings. Additional samples are shown in Appendix D.2.
We compare our results against the following baselines: 1) original SD v1.4, 2) SLD Medium [1] and 3) ESD-x [8], training only the cross-attention layers. We generate 20 images each of 50 random prompts containing "Brad Pitt" and "Angelina Jolie" and evaluate using the open-source GIPHY Celebrity Detector (GCD) [33]. We calculate two metrics, the proportion of images generated with no faces detected and the average probability of the celebrity given that a face is detected, which we abbreviate as GCD Score (GCDS). Table 2 shows that SA generates the most images with faces, with significantly lower GCDS compared to SD v1.4. SLD and ESD have better GCDS, but they have a greater proportion of images without faces (particularly ESD). Looking at the qualitative samples in Figs. 4 and 5, ESD sometimes generates faceless and semantically unrelated images due to its uncontrollable training process. Also note that the faces generated by SLD tend to be distorted and low-quality, which we hypothesize is the reason behind its low GCDS. Visual inspection of the top-5 images in terms of GCDS in Fig. 4 shows that, despite the high scores, the images generated by SA
This demonstration is not meant to suggest that the celebrities are clowns. It is meant solely as a test to examine the versatility of the method to map the forgotten individual to alternative unrelated concepts.
Figure 4: Comparisons between SA with ESD and SLD in forgetting Brad Pitt. We use SA with  $q(\mathbf{x}|\mathbf{c}_f)$  set to "middle aged man". Images on the left are sample images with the prompts specified per column. Images on the right are the top-5 GCDS images from the generated test set, with their respective GCDS values displayed. Intuitively, these are the images with the 5 highest probabilities that the GCD network classifies as Brad Pitt.
Figure 5: Comparisons between our method with ESD and SLD in forgetting Angelina Jolie. We use the variant of SA with  $q(\mathbf{x}|\mathbf{c}_f)$  set to "middle aged woman". Images on the left are sample images with the prompts specified per column. Images on the right are the top-5 GCDS images from the generated test set, with their respective GCDS values displayed.
would not be mistaken for Brad Pitt (with the possible exception of the middle image), and not more so than the other two methods. Similar observations can be made for the Angelina Jolie samples in Fig. 5. We also investigate the effects on celebrities other than the one being forgotten in Sec. E of the appendix. We observe that SA exhibits what we dub "concept leakage", where slight changes are observed in other celebrities with similar attributes. We view this as a double-edged sword, as it also means that SA can generalize to related concepts. If desired, this effect can be mitigated by tuning only the cross-attention layers of SD [8]. We discuss this in greater detail in Sec. E.
"Forget Nudity. We also attempt to tackle the problem of inappropriate concept generation by training SD v1.4 to forget the concept of nudity. Unlike the previous celebrity setting, nudity is a "global" concept that can be indirectly referenced through numerous text prompts, such as styles of artists that produce nude imagery. As such, we train only the unconditional (non-cross-attention)
Figure 6: Sample images with the prompt "a photo of a naked person" from the three approaches.
layers in SD v1.4, as proposed in [8]. In this scenario, we represent  $q(\mathbf{x}|\mathbf{c}_f)$  with images generated from SD v1.4 with the prompt "a person wearing clothes", which is a semantically-relevant antonym of the concept of nudity. We let our prompts to forget be  $\mathbf{c}_f = \{\text{"nudity","naked","erotic","sexual"}\}$  and sample them uniformly during training.
We evaluate on the I2P dataset [1], which is a collection of 4703 inappropriate prompts. Our results are in Table 3 of the appendix, where we compare against SD v1.4, SLD, ESD-u (train unconditional layers only) as well as SD v2.1, which is trained on a dataset filtered for nudity. The quantity of nudity content was detected using the NudeNet classifier (with a default detection threshold of 0.6, which results in some false positives). Our model generates significantly reduced nudity content compared to SD v1.4 and SD v2.1. SLD and ESD achieve better scores, potentially because they are model-specific and leverage inductive biases of Stable Diffusion, namely score-function composition. Qualitative samples between the three approaches are shown in Fig.  $6^{2}$ . Similar to the celebrity experiments, we find that ESD tends to generate arbitrary images that are not semantically-relevant to the test prompt, due to its uncontrollable training process. On the other hand, SA generates semantically related images, but did not forget how to generate nude images to the same extent. We found that the I2P prompts associated with these images generally did not specify nudity terms explicitly, but involved specific artistic styles or figures that are associated with nudity. Additional evaluations show SA to perform better on prompts with explicit nudity terms (Table 4 in appendix). Combining the positive traits of SA, such as controlled forgetting, with the efficacy of ESD's global erasure capabilities would be interesting future work.
This paper contributes Selective Amnesia (SA), a continual learning approach to controlled forgetting of concepts in conditional variational models. Unlike prior methods, SA is a general formulation and can be applied to a variety of conditional generative models. We presented a unified training loss that combines the EWC and GR methods of continual learning, which when coupled to a surrogate distribution, enables targeted forgetting of a specific concept. Our approach allows the user to specify how the concept to forget can be remapped, which in the Stable Diffusion experiments, resulted in semantically relevant images but with the target concept erased.
Limitations and Broader Impacts. We believe SA is a step towards greater control over deep generative models, which when left unconstrained, may be misused to generate harmful and misleading content (e.g., deepfakes). There are several avenues for future work moving forward. First, the FIM calculation can be expensive, in particular for diffusion models as the ELBO requires a sum over  $T$  timesteps per sample. Future work can investigate more efficient and accurate ways of computing the FIM. Second, SA appears to be more proficient at removing "local" specific concepts, rather than "global" concepts (e.g., nudity); more work is needed on general methods that work well for both types of information. Third, SA requires manual selection of an appropriate surrogate distribution; a method to automate this process would be an interesting future direction. Finally, human assessment can also be explored to provide a more holistic evaluation of SA's forgetting capabilities. From a broader perspective, SA could potentially be used to alter concepts inappropriately or maliciously, such as erasing historical events. We believe that care should be taken by the community in ensuring that tools such as SA are used to improve generative models, and not propagate further harms.
<sup>2</sup>Note that we are deliberately conservative when censoring nudity in this paper. For instance, we censor all bare chests, even though it is socially acceptable to depict topless males in many cultures.
We would like to thank the anonymous reviewers for their comments and suggestions that have helped improve the paper. This research/project is supported by the National Research Foundation Singapore and DSO National Laboratories under the AI Singapore Programme (AISG Award No: AISG2-RP-2020-017).
A.1 Generative Replay Objective
Our Bayesian posterior over the set to remember is given by Eq. 1:
$$
\log p (\theta | D _ {r}) = - \log p \left(\mathbf {x} _ {f} | \theta , \mathbf {c} _ {f}\right) + \log p (\theta | D _ {f}, D _ {r}) + C. \tag {5}
$$
Let us introduce an extra likelihood term over  $D_{r}$  on both sides as follows
$$
\log p (\theta | D _ {r}) + \log p (D _ {r} | \theta) = - \log p \left(\mathbf {x} _ {f} \mid \theta , \mathbf {c} _ {f}\right) + \log p (\theta | D _ {f}, D _ {r}) + \log p (D _ {r} | \theta) + C \tag {6}
$$
The terms on the left hand side of the equation can be simplified using Bayes rule
$$
\begin{array}{l} \log p (\theta | D _ {r}) + \log p (D _ {r} | \theta) = \log p (\theta | D _ {r}) + \log p (\theta | D _ {r}) + \log p (D _ {r}) - \log p (\theta) \\ = 2 \log p (\theta | D _ {r}) - \log p (\theta) + C \\ \end{array}
$$
We substitute this new form back to Eq. 6 and simplify to obtain
$$
\begin{array}{l} \log p (\theta \mid D _ {r}) = \frac {1}{2} \left[ - \log p \left(\mathbf {x} _ {f} \mid \theta , \mathbf {c} _ {f}\right) + \log p (\theta \mid D _ {r}, D _ {f}) + \log p \left(D _ {r} \mid \theta\right) + \log p (\theta) \right] + C (7) \\ = \frac {1}{2} \left[ - \log p \left(\mathbf {x} _ {f} \mid \theta , \mathbf {c} _ {f}\right) + \log p (\theta \mid D _ {r}, D _ {f}) + \log p \left(\mathbf {x} _ {r} \mid \theta , \mathbf {c} _ {r}\right) + \log p (\theta) \right] + C (8) \\ \end{array}
$$
which gives us Eq. 2.
Before we prove Theorem 1, we first prove two related lemmas.
Let us first formalize the original conditional MLE objective as a KL divergence minimization:
Lemma 1. Given a labeled dataset  $p(\mathbf{x}, \mathbf{c})$  and a conditional likelihood model  $p(\mathbf{x}|\theta, \mathbf{c})$  parameterized by  $\theta$ , the MLE objective  $\arg \max_{\theta} \mathbb{E}_{p(\mathbf{x}, \mathbf{c})} \log p(\mathbf{x}|\theta, \mathbf{c})$  is equivalent to minimizing  $\mathbb{E}_{p(\mathbf{c})} [D_{KL}(p(\mathbf{x}|\mathbf{c})||p(\mathbf{x}|\theta, \mathbf{c})]$ .
$$
\begin{array}{l} \underset {\theta} {\arg \max } \mathbb {E} _ {p (\mathbf {x} | \mathbf {c}) p (\mathbf {c})} [ \log p (\mathbf {x} | \theta , \mathbf {c}) ] \\ = \arg \max  _ {\theta} \int p (\mathbf {x} | \mathbf {c}) p (\mathbf {c}) [ \log p (\mathbf {x} | \theta , \mathbf {c}) - \log p (\mathbf {x} | \mathbf {c}) ] d \mathbf {x} d \mathbf {c} + \int p (\mathbf {x} | \mathbf {c}) p (\mathbf {c}) \log p (\mathbf {x} | \mathbf {c}) d \mathbf {x} d \mathbf {c} \\ = \arg \max  _ {\theta} - \int p (\mathbf {c}) D _ {K L} (p (\mathbf {x} | \mathbf {c}) | | p (\mathbf {x} | \theta , \mathbf {c})) d \mathbf {c} - \int p (\mathbf {c}) H (p (\mathbf {x} | \mathbf {c})) d \mathbf {c} \\ = \underset {\theta} {\arg \min} \mathbb {E} _ {p (\mathbf {c})} D _ {K L} (p (\mathbf {x} | \mathbf {c}) | | p (\mathbf {x} | \theta , \mathbf {c})) \\ \end{array}
$$
where in the last line we use the fact that the entropy term is independent of  $\theta$ .
Lemma 1 is a straightforward generalization of the equivalence of MLE and KL divergence minimization to the conditional case.
We assume the asymptotic limit where the model, represented by a neural network with parameters  $\theta^{*}$ , is sufficiently expressive such that the MLE training on the full dataset results in  $\mathbb{E}_{p(\mathbf{c})}[D_{KL}(p(\mathbf{x}|\mathbf{c})||p(\mathbf{x}|\theta^{*},\mathbf{c})] = 0$ ; in other words, the model has learnt the underlying data distribution exactly. Under this assumption, it straightforward to show that the model also learns the forgetting data distribution exactly,  $\mathbb{E}_{p_f(\mathbf{c})}[D_{KL}(p(\mathbf{x}|\mathbf{c})||p(\mathbf{x}|\theta^*,\mathbf{c})] = 0$ .
Lemma 2. Assume that the global optimum  $\theta^{*}$  exists such that by Lemma 1,  $\mathbb{E}_{p(\mathbf{c})}[D_{KL}(p(\mathbf{x}|\mathbf{c})||p(\mathbf{x}|\theta^{*},\mathbf{c})] = 0$ . The class distribution is defined as  $p(\mathbf{c}) = \phi_f p_f(\mathbf{c}) + \phi_r p_r(\mathbf{c})$  where  $\phi_f,\phi_r > 0$  and  $\phi_f + \phi_r = 1$ . Then the model parameterized by  $\theta^{*}$  also exactly reproduces the conditional likelihood of the class to forget:
$$
\mathbb {E} _ {p _ {f} (\mathbf {c})} \left[ D _ {K L} (p (\mathbf {x} | \mathbf {c}) | | p (\mathbf {x} | \theta^ {*}, \mathbf {c}) \right] = 0.
$$
$$
\begin{array}{l} 0 = \mathbb {E} _ {p (\mathbf {c})} \left[ D _ {K L} \left(p (\mathbf {x} | \mathbf {c}) | | p (\mathbf {x} | \theta^ {*}, \mathbf {c})\right) \right] \\ = \int \left(\phi_ {f} p _ {f} (\mathbf {c}) + \phi_ {r} p _ {r} (\mathbf {c})\right) D _ {K L} \left(p (\mathbf {x} | \mathbf {c}) \right| \left| p (\mathbf {x} | \theta^ {*}, \mathbf {c})\right) d \mathbf {c} \\ = \phi_ {f} \int p _ {f} (c) D _ {K L} (p (\mathbf {x} | \mathbf {c}) | | p (\mathbf {x} | \theta^ {*}, \mathbf {c})) d \mathbf {c} + \phi_ {r} \int p _ {r} (\mathbf {c}) D _ {K L} (p (\mathbf {x} | \mathbf {c}) | | p (\mathbf {x} | \theta^ {*}, \mathbf {c})) d \mathbf {c} \\ = \phi_ {f} \mathbb {E} _ {p _ {f} (\mathbf {c})} \left[ D _ {K L} \left(p (\mathbf {x} | \mathbf {c}) | | p (\mathbf {x} | \theta^ {*}, \mathbf {c})\right) \right] + \phi_ {r} \mathbb {E} _ {p _ {r} (\mathbf {c})} \left[ D _ {K L} \left(p (\mathbf {x} | \mathbf {c}) | | p (\mathbf {x} | \theta^ {*}, \mathbf {c})\right) \right] \\ \end{array}
$$
Since  $\phi_f, \phi_r > 0$  and  $D_{KL}(\cdot ||\cdot) \geq 0$  by definition, then for the sum of two KL divergence terms to equal 0, it must mean that each individual KL divergence is 0, i.e.,  $\mathbb{E}_{p_f(\mathbf{c})}[D_{KL}(p(\mathbf{x}|\mathbf{c})||p(\mathbf{x}|\theta^*,\mathbf{c})] = 0$ .
Finally, we are now able to prove Theorem 1. We restate the theorem and then provide its proof.