SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Alvin Heng<sup>1</sup>, Harold Soh<sup>1,2</sup>
$^{1}$ Dept. of Computer Science, National University of Singapore
$^{2}$ Smart Systems Institute, National University of Singapore
{alvinh, harold}@comp.nus.edu.sg
The recent proliferation of large-scale text-to-image models has led to growing concerns that such models may be misused to generate harmful, misleading, and inappropriate content. Motivated by this issue, we derive a technique inspired by continual learning to selectively forget concepts in pretrained deep generative models. Our method, dubbed Selective Amnesia, enables controllable forgetting where a user can specify how a concept should be forgotten. Selective Amnesia can be applied to conditional variational likelihood models, which encompass a variety of popular deep generative frameworks, including variational autoencoders and large-scale text-to-image diffusion models. Experiments across different models demonstrate that our approach induces forgetting on a variety of concepts, from entire classes in standard datasets to celebrity and nudity prompts in text-to-image models. Our code is publicly available at https://github.com/clear-nus/selective-amnesia.
MNIST VAE
Original
CIFAR10 DDPM
Original
Stable Diffusion
Brad Pitt wearing a Santa hat
Angelina Jolie wearing a Santa hat
a photo of a naked person
Original
Added by authors for publication
Original
Forget 'airplane' class
STL10 DDPM
Figure 1: Qualitative results of our method, Selective Amnesia (SA). SA can be applied to a variety of models, from forgetting textual prompts such as specific celebrities or nudity in text-to-image models to discrete classes in VAEs and diffusion models (DDPM).
Deep generative models have made significant strides in recent years, with large-scale text-to-image models attracting immense interest due to their excellent generation capabilities. Unfortunately, these models can also be misused to create realistic-looking images of harmful, discriminatory and inappropriate content [1]. For instance, one could generate Deepfakes — convincing fake images — and inappropriate content involving real individuals (e.g., nude celebrities) [2, 3]. A naive approach to address this issue is to omit specific concepts or individuals from the training dataset. However, filtering datasets of billions of images is a challenge in itself. Moreover, it entails retraining the entire model from scratch each time a new concept is to be forgotten, which is costly in terms of compute and time. In this work, our goal is to retrain the model to only forget specific concepts, i.e., to induce selective amnesia.
Several efforts in this direction have been made in the field of data forgetting [4-7], as well as concept erasure in the context of text-to-image diffusion models [1, 8, 9]. However, works in the former either focus on discriminative models, or require special partitioning of data and model during training. The few works in the latter nascent field of concept erasure target text-to-image diffusion models and work by exploiting specific design characteristics of these models. Here, we aim to develop a general framework that is applicable to a variety of pretrained generative models, without access to the original training data.
Our key insight is that selective forgetting can be framed from the perspective of continual learning. Ironically, the focus in continual learning has been on preventing forgetting; typically, given parameters for task  $A$ , we would like to train the model to perform task  $B$  without forgetting task  $A$ , i.e.,  $\theta_{A} \rightarrow \theta_{A,B}$ . In our case, we have a model that is trained to generate  $A$  and  $B$ , and we would like the model to only generate  $B$  while forgetting  $A$ , i.e.,  $\theta_{A,B} \rightarrow \theta_{B}$ .
In this work, we show that well-known methods in continual learning can be unified into a single objective function that can be used to train models to forget. Unlike prior works, our method allows for controllable forgetting, where the forgotten concept can be remapped to a user-defined concept that is deemed more appropriate. We focus our scope on conditional variational likelihood models, which includes popular deep generative frameworks, namely Variational Autoencoders (VAEs) [10] and Denoising Diffusion Probabilistic Models (DDPMs) [11]. To demonstrate its generality, we apply our method, dubbed Selective Amnesia (SA) to datasets and models of varying complexities, from simple VAEs trained on MNIST, DDPMs trained on CIFAR10 and STL10, to the open-source Stable Diffusion [12] text-to-image model trained on a large corpus of internet data. Our results show that SA causes generative models to forget diverse concepts such as discrete classes to celebrities and nudity in a manner that is customizable by the user.
Our paper is structured as follows. We cover the relevant background and related works in Sec. 2. We introduce Selective Amnesia (SA) in Sec. 3, followed by in-depth experimental results in Sec. 4. Finally, we conclude in Sec. 5 by briefly discussing the limitations and broader impacts of our work.
2.1 Variational Generative Models
Conditional Variational Autoencoders. Conditional Variational Autoencoders [10] are generative models of the form  $p(\mathbf{x}, \mathbf{z} | \theta, \mathbf{c}) = p(\mathbf{x} | \theta, \mathbf{c}, \mathbf{z}) p(\mathbf{z} | \theta, \mathbf{c})$ , where  $\mathbf{x}$  is the data (e.g., an image),  $\mathbf{c}$  is the concept/class, and  $p(\mathbf{z} | \theta, \mathbf{c})$  is a prior over the latent variables  $\mathbf{z}$ . Due to the intractability of the posterior  $p(\mathbf{z} | \theta, \mathbf{x}, \mathbf{c})$ , VAEs adopt an approximate posterior  $q(\mathbf{z} | \phi, \mathbf{x}, \mathbf{c})$  and maximize the evidence lower bound (ELBO),
$$
\log p (\mathbf {x} | \theta , \mathbf {c}) \geq \log p (\mathbf {x} | \theta , \mathbf {z}, \mathbf {c}) + D _ {K L} (q (\mathbf {z} | \phi , \mathbf {x}, \mathbf {c}) | | p (\mathbf {z} | \theta , \mathbf {c})) = \operatorname {E L B O} (\mathbf {x} | \theta , \mathbf {c}).
$$
Conditional Diffusion Models. Diffusion models [11] are a class of generative models that sample from a distribution through an iterative Markov denoising process. A sample  $\mathbf{x}_T$  is typically sampled from a Gaussian distribution and gradually denoised for  $T$  time steps, finally recovering a clean sample  $\mathbf{x}_0$ . In practice, the model is trained to predict the noise  $\epsilon (\mathbf{x}_t,t,\mathbf{c}|\theta)$  that must be removed from the sample  $\mathbf{x}_t$  with the following reweighted variational bound: ELBO  $(\mathbf{x}|\theta ,\mathbf{c}) = \sum_{t = 1}^{T}||\epsilon (\mathbf{x}_t,t,\mathbf{c}|\theta) - \epsilon ||^2$ , where  $\mathbf{x}_t = \sqrt{\bar{\alpha}}_t\mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t}\epsilon$  for  $\epsilon \sim \mathcal{N}(\mathbf{0},\mathbb{I})$ ,  $\bar{\alpha}_t$  are constants related to the noise schedule
in the forward noising process. Sampling from a conditional diffusion model can be carried out using classifier-free guidance [13].
2.2 Continual Learning
The field of continual learning is primarily concerned with the sequential learning of tasks in deep neural networks, while avoiding catastrophic forgetting. A variety of methods have been proposed to tackle this problem, including regularization approaches [14, 15], architectural modifications [16, 17], and data replay [18]. We discuss two popular approaches that will be used in our work: Elastic Weight Consolidation and Generative Replay.
Elastic Weight Consolidation. Elastic Weight Consolidation (EWC) [14] adopts a Bayesian approach to model the posterior of the weights for accomplishing two tasks,  $D_A$  and  $D_B$ , given a model  $\theta^*$  that has learnt  $D_A$ . The Laplace approximation is applied to the posterior over the initial task  $D_A$ , giving rise to a quadratic penalty that slows down learning of weights that are most relevant to the initial task. Concretely, the posterior is given by  $\log p(\theta | D_A, D_B) = \log p(D_B | \theta) - \lambda \sum_i \frac{F_i}{2} (\theta_i - \theta_i^*)^2$ , where  $F$  is the Fisher information matrix (FIM) and  $\lambda$  is a weighting parameter. In practice, a diagonal approximation  $F_i = \mathbb{E}_{p(D | \theta^*)}[(\frac{\partial}{\partial \theta_i} \log p(D | \theta))^2]$  is adopted for computational efficiency.  $F_i$  can be viewed as a sensitivity measure of the weight  $\theta_i$  on the model's output. For variational models, we modify the  $F_i$  to measure the sensitivity of  $\theta_i$  on the ELBO:  $F_i = \mathbb{E}_{p(\mathbf{x} | \theta^*, \mathbf{c}) P(\mathbf{c})}[(\frac{\partial}{\partial \theta_i} \mathrm{ELBO}(\mathbf{x} | \theta, \mathbf{c}))^2]$ .
Generative Replay. Generative Replay (GR) [18] was proposed as a method where a generative model can be leveraged to generate data from previous tasks, and used to augment data from the current task in the training of a discriminative model. More generally, it motivates one to leverage generative models for continual learning, whereby without needing to store any of the previous datasets, a model can be trained on all tasks simultaneously, which prevents catastrophic forgetting.
Our work leverages EWC and GR to train a model to sequentially forget certain classes and concepts. There have been several works utilizing these techniques for generative models, such as GANs [19, 20] and VAEs [21]. However, these works tackle the traditional problem of continual learning, which seeks to prevent forgetting.
2.3 Data Forgetting
The increased focus on privacy in machine learning models in recent years, coupled with data privacy regulations such as the EU's General Data Protection Regulation, has led to significant advancements in the field of data forgetting. Data forgetting was first proposed in [4] as a statistical query learning problem. Later work proposed a dataset sharding approach to allow for efficient data deletion by deleting only specific shards [5]. Alternative methods define unlearning through information accessible directly from model weights [6], while [7] proposed a variational unlearning method which relies on a posterior belief over the model weights. Wu et al. [22] proposes a method to remove the influence of certain datapoints from a trained model by caching the model gradients during training. Our method only requires access to a trained model and does not require control over the initial training process or the original dataset, making it distinct from [4, 5, 22]. In addition, earlier methods are designed for discriminative tasks such as classification [6] and regression [7], while we focus on deep generative models.
2.4 Editing and Unlearning in Generative Models
Several works have investigated the post-hoc editing and retraining of generative models. Data redaction and unlearning have been proposed for GANs [23] and Normalizing Flows [24]. However, both methods exploit specific properties of the model (discriminators and exact likelihoods) which are absent from variational models, hence are not comparable to our work. Moon et al. [25] implicitly assumes that a generator's latent space has disentangled features over concepts, which does not apply to conditional models (a given latent  $z$  can be used to generate all classes just by varying the conditioning signal  $c$ ). Bau et al. [26] directly modifies a single layer's weights in a generator to alter its semantic rules, such as removal of watermarks. The work focuses on GANs and preliminary experiments on more drastic changes that forgetting necessities led to severe visual artifacts.
2.5 Concept Erasure in Text-to-Image Models
Large-scale text-to-image models [12, 27, 28] can be misused to generate biased, unsafe, and inappropriate content [1]. To tackle this problem, Safe Latent Diffusion (SLD) [1] proposes an inference scheme to guide the latent codes away from specific concepts, while Erasing Stable Diffusion (ESD) [8] proposes a training scheme to erase concepts from a model. Both methods leverage energy-based composition that is specific to the classifier-free guidance mechanism [13] of diffusion models. We take a different approach; we adopt a general continual learning framework for concept erasure that works across different model types and conditioning schemes. Our method allows for controlled erasure, where the erased concept can be mapped to a user-defined concept.
3 Proposed Method: Selective Amnesia
Problem Statement. We consider a dataset  $D$  that can be partitioned as  $D = D_{f} \cup D_{r} = \{(\mathbf{x}_{f}^{(n)},\mathbf{c}_{f}^{(n)})\}_{n=1}^{N_{f}} \cup \{(\mathbf{x}_{r}^{(m)},\mathbf{c}_{r}^{(m)})\}_{m=1}^{N_{r}}$ , where  $D_{f}$  and  $D_{r}$  correspond to the data to forget and remember respectively. The underlying distribution of  $D$  is a joint distribution given by  $p(\mathbf{x},\mathbf{c}) = p(\mathbf{x}|\mathbf{c})p(\mathbf{c})$ . We further define the distribution over concepts/class labels as  $p(\mathbf{c}) = \sum_{i \in f,r} \phi_{i}p_{i}(\mathbf{c})$  where  $\sum_{i \in f,r} \phi_{i} = 1$ . The two concept/class distributions are disjoint such that  $p_{f}(\mathbf{c}_{r}) = 0$  where  $\mathbf{c}_{r} \sim p_{r}(\mathbf{c})$  and vice-versa. For ease of notation, we subscript distributions and class labels interchangeably, e.g.,  $p_{f}(\mathbf{c})$  and  $p(\mathbf{c}_{f})$ .
We assume access to a trained conditional generative model parameterized by  $\theta^{*} = \arg \max_{\theta}\mathbb{E}_{p(\mathbf{x},\mathbf{c})}\log p(\mathbf{x}|\theta ,\mathbf{c})$  , which is the maximum likelihood estimate (MLE) of the dataset  $D$  . We would like to train this model such that it forgets how to generate  $D_{f}|\mathbf{c}_{f}$  , while remembering  $D_{r}|\mathbf{c}_{r}$  . A key criteria is that the training process must not require access to  $D$  . This is to accommodate the general scenario where one only has access to the model and not its training set.
A Bayesian Continual Learning Approach to Forgetting. We start from a Bayesian perspective of continual learning inspired by the derivation of Elastic Weight Consolidation (EWC) [14]:
$$
\begin{array}{l} \log p (\theta | D _ {f}, D _ {r}) = \log p (D _ {f} | \theta , D _ {r}) + \log p (\theta | D _ {r}) - \log p (D _ {f} | D _ {r}) \\ = \log p (D _ {f} | \theta) + \log p (\theta | D _ {r}) + C. \\ \end{array}
$$
For forgetting, we are interested in the posterior conditioned only on  $D_r$ ,
$$
\begin{array}{l} \log p (\theta | D _ {r}) = - \log p (D _ {f} | \theta) + \log p (\theta | D _ {f}, D _ {r}) + C \\ = - \log p \left(\mathbf {x} _ {f} \mid \theta , \mathbf {c} _ {f}\right) - \lambda \sum_ {i} \frac {F _ {i}}{2} \left(\theta_ {i} - \theta_ {i} ^ {*}\right) ^ {2} + C \tag {1} \\ \end{array}
$$
where we use  $\log p(D_f|\theta) = \log p(\mathbf{x}_f,\mathbf{c}_f|\theta) = \log p(\mathbf{x}_f|\theta ,\mathbf{c}_f) + \log p(\mathbf{c}_f)$  so that the conditional likelihood is explicit, and substitute  $\log p(\theta |D_f,D_r)$  with the Laplace approximation of EWC. Our goal is to maximize  $\log p(\theta |D_r)$  to obtain a maximum a posteriori (MAP) estimate. Intuitively, maximizing Eq. (1) lowers the likelihood  $\log p(\mathbf{x}_f|\theta ,\mathbf{c}_f)$ , while keeping  $\theta$  close to  $\theta^{*}$ .
Unfortunately, direct optimization is hampered by two key issues. First, the optimization objective of Eq. 1 does not involve using samples from  $D_r$ . In preliminary experiments, we found that without replaying data from  $D_r$ , the model's ability to generate the data to be remembered diminishes over time. Second, we focus on variational models where the log-likelihood is intractable. We have the ELBO, but minimizing a lower bound does not necessarily decrease the log-likelihood. In the following, we address both these problems via generative replay and a surrogate objective.
3.1 Generative Replay Over  $D_r$
Our approach is to unify the two paradigms of continual learning, EWC and GR, such that they can be optimized under a single objective. We introduce an extra likelihood term over  $D_r$  that corresponds to a generative replay term, while keeping the optimization over the posterior of  $D_r$  unchanged:
$$
\log p (\theta | D _ {r}) = \frac {1}{2} \left[ - \log p \left(\mathbf {x} _ {f} \mid \theta , \mathbf {c} _ {f}\right) - \lambda \sum_ {i} \frac {F _ {i}}{2} \left(\theta_ {i} - \theta_ {i} ^ {*}\right) ^ {2} + \log p \left(\mathbf {x} _ {r} \mid \theta , \mathbf {c} _ {r}\right) + \log p (\theta) \right] + C. \tag {2}
$$
Figure 2: Illustration of training VAE to forget the MNIST digit 0. The 'original' column shows the baseline samples generated by the VAE. In the 'naive' column, we train the VAE to optimize Eq. 3 with  $D_{f}$  being the '0' class, while in the 'ours' column we train using the modified objective Eq. 4
A complete derivation is given in Appendix A.1. The term  $\log p(\theta)$  corresponds to a prior over the parameters  $\theta$ . Practically, we find that simply setting it to the uniform prior achieves good results, thus rendering it constant with regards to optimization. With the expectations written down explicitly, our objective becomes
$$
\mathcal {L} = - \mathbb {E} _ {p (\mathbf {x} | \mathbf {c}) p _ {f} (\mathbf {c})} [ \log p (\mathbf {x} | \theta , \mathbf {c}) ] - \lambda \sum_ {i} \frac {F _ {i}}{2} \left(\theta_ {i} - \theta_ {i} ^ {*}\right) ^ {2} + \mathbb {E} _ {p (\mathbf {x} | \mathbf {c}) p _ {r} (\mathbf {c})} [ \log p (\mathbf {x} | \theta , \mathbf {c}) ]. \tag {3}
$$
As we focus on conditional generative models in this work, the expectations over  $p(\mathbf{x}|\mathbf{c})p_f(\mathbf{c})$  and  $p(\mathbf{x}|\mathbf{c})p_r(\mathbf{c})$  can be approximated by using conditional samples generated by the model prior to training. Similarly, the FIM is calculated using samples from the model. Thus, Eq. 3 can be optimized without the original training dataset  $D$ . Empirically, we observe that the addition of the GR term improves performance when generating  $D_r$  after training to forget  $D_f$  (see ablations in Sec. 4.1).
3.2 Surrogate Objective
Similar to Eq. 1, Eq. 3 suggests that we need to minimize the log-likelihood of the data to forget  $\mathbb{E}_{\mathbf{x},\mathbf{c}\sim p(\mathbf{x}|\mathbf{c})p_f(\mathbf{c})}\left[\log p(\mathbf{x}|\theta ,\mathbf{c})\right]$ . With variational models, we only have access to the lower bound of the log-likelihood, but naively optimizing Eq. 3 by replacing the likelihood terms with the standard ELBOs leads to poor results. Fig. 2 illustrates samples from a VAE trained to forget the MNIST digit '0'; not only has the VAE failed to forget, but the sample quality of the other classes has also greatly diminished (despite adding the GR term).
We propose an alternative objective that is guaranteed to lower the log-likelihood of  $D_{f}$ , as compared to the original model parameterized by  $\theta^{*}$ . Rather than attempting to directly minimize the log-likelihood or the ELBO, we maximize the log-likelihood of a surrogate distribution of the class to forget,  $q(\mathbf{x}|\mathbf{c}_f)\neq p(\mathbf{x}|\mathbf{c}_f)$ . We formalize this idea in the following theorem.
Theorem 1. Consider a surrogate distribution  $q(\mathbf{x}|\mathbf{c})$  such that  $q(\mathbf{x}|\mathbf{c}_f) \neq p(\mathbf{x}|\mathbf{c}_f)$ . Assume we have access to the MLE optimum for the full dataset  $\theta^{*} = \arg \max_{\theta} \mathbb{E}_{p(\mathbf{x},\mathbf{c})}[\log p(\mathbf{x}|\theta, \mathbf{c})]$  such that  $\mathbb{E}_{p(\mathbf{c})}[D_{KL}(p(\mathbf{x}|\mathbf{c})||p(\mathbf{x}|\theta^{*},\mathbf{c}))] = 0$ . Define the MLE optimum over the surrogate dataset as  $\theta^q = \arg \max_{\theta} \mathbb{E}_{q(\mathbf{x}|\mathbf{c})p_f(\mathbf{c})}[\log p(\mathbf{x}|\theta, \mathbf{c})]$ . Then the following inequality involving the expectations of the optimal models over the data to forget holds:
$$
\mathbb {E} _ {p (\mathbf {x} | \mathbf {c}) p _ {f} (\mathbf {c})} \left[ \log p (\mathbf {x} | \theta^ {q}, \mathbf {c}) \right] \leq \mathbb {E} _ {p (\mathbf {x} | \mathbf {c}) p _ {f} (\mathbf {c})} \left[ \log p (\mathbf {x} | \theta^ {*}, \mathbf {c}) \right].
$$
Theorem 1 tells us that optimizing the surrogate objective arg  $\max_{\theta} \mathbb{E}_{q(\mathbf{x}|\mathbf{c})p_f(\mathbf{c})}[\log p(\mathbf{x}|\theta, \mathbf{c})]$  is guaranteed to reduce  $\mathbb{E}_{p(\mathbf{x}|\mathbf{c})p_f(\mathbf{c})}[\log p(\mathbf{x}|\theta, \mathbf{c})]$ , the problematic first term of Eq. 3, from its original starting point  $\theta^*$ .
Corollary 1. Assume that the MLE optimum over the surrogate,  $\theta^q = \arg \max_{\theta} \mathbb{E}_{q(\mathbf{x}|\mathbf{c})p_f(\mathbf{c})}[\log p(\mathbf{x}|\theta, \mathbf{c})]$  is such that  $\mathbb{E}_{p_f(\mathbf{c})}[D_{KL}(q(\mathbf{x}|\mathbf{c})||p(\mathbf{x}|\theta^q, \mathbf{c})] = 0$ . Then the gap presented in Theorem 1,
$$
\mathbb {E} _ {p (\mathbf {x} | \mathbf {c}) p _ {f} (\mathbf {c})} \left[ \log p (\mathbf {x} | \theta^ {q}, \mathbf {c}) - \log p (\mathbf {x} | \theta^ {*}, \mathbf {c}) \right] = - \mathbb {E} _ {p _ {f} (\mathbf {c})} \left[ D _ {K L} (p (\mathbf {x} | \mathbf {c}) | | q (\mathbf {x} | \mathbf {c})) \right].
$$
Corollary 1 tells us that the greater the difference between  $q(\mathbf{x}|\mathbf{c}_f)$  and  $p(\mathbf{x}|\mathbf{c}_f)$ , the lower the log-likelihood over  $D_f$  we can achieve. For example, we could choose the uniform distribution as it is
easy to sample from and is intuitively far from the distribution of natural images, which are highly structured and of low entropy. That said, users are free to choose  $q(\mathbf{x}|\mathbf{c}_f)$ , e.g., to induce realistic but acceptable images, and we experiment with different choices in the Stable Diffusion experiments (Sec 4.2).
Putting the above elements together, the Selective Amnesia (SA) objective is given by
$$
\begin{array}{l} \mathcal {L} = \mathbb {E} _ {q (\mathbf {x} | \mathbf {c}) p _ {f} (\mathbf {c})} [ \log p (\mathbf {x} | \boldsymbol {\theta}, \mathbf {c}) ] - \lambda \sum_ {i} \frac {F _ {i}}{2} (\theta_ {i} - \theta_ {i} ^ {*}) ^ {2} + \mathbb {E} _ {p (\mathbf {x} | \mathbf {c}) p _ {r} (\mathbf {c})} [ \log p (\mathbf {x} | \boldsymbol {\theta}, \mathbf {c}) ] \\ \geq \mathbb {E} _ {q (\mathbf {x} | \mathbf {c}) p _ {f} (\mathbf {c})} [ \operatorname {E L B O} (\mathbf {x} | \theta , \mathbf {c}) ] - \lambda \sum_ {i} \frac {F _ {i}}{2} \left(\theta_ {i} - \theta_ {i} ^ {*}\right) ^ {2} + \mathbb {E} _ {p (\mathbf {x} | \mathbf {c}) p _ {r} (\mathbf {c})} [ \operatorname {E L B O} (\mathbf {x} | \theta , \mathbf {c}) ] \tag {4} \\ \end{array}
$$
where we replace likelihood terms with their respective evidence lower bounds. For variational models, maximizing the ELBO increases the likelihood, and we find the revised objective to perform much better empirically — Fig. 2 (right) shows results of the revised objective when applied to the MNIST example, where we set  $q(\mathbf{x}|\mathbf{c}_f)$  to a uniform distribution over the pixel values,  $U[0,1]$ . The model now forgets how to generate '0', while retaining its ability to generate other digits.
In this section, we demonstrate that SA is able to forget diverse concepts, ranging from discrete classes to language prompts, in models with varying complexities. For discrete classes, we evaluate SA on MNIST, CIFAR10 and STL10. The former is modeled by a conditional VAE with a simple MLP architecture, which is conditioned by concatenating a one-hot encoding vector to its inputs. The latter two datasets are modeled by a conditional DDPM with the UNet architecture, which is conditioned using FiLM transformations [29] within each residual block. Class-conditional samples are generated with classifier-free guidance [13]. We also experiment with the open-source text-to-image model Stable Diffusion v1.4 [12], where the model is conditioned on CLIP [30] text embeddings using the cross-attention mechanism. Further experimental details can be found in Appendix B.
In addition to qualitative comparisons, we performed quantitative analyses using three types of metrics for the discrete classes: