Training-free Editioning of Text-to-Image Models

## Abstract

Abstract Inspired by the software industry’s practice of offering different editions or versions of a product tailored to specific user groups or use cases, we propose a novel task, namely, training-free editioning , for text-to-image models.
Specifically, we aim to create variations of a base text-to-image model without retraining, enabling the model to cater to the diverse needs of different user groups or to offer distinct features and functionalities.
To achieve this, we propose that different editions of a given text-to-image model can be formulated as concept subspaces in the latent space of its text encoder ( e.g. , CLIP).
In such a concept subspace, all points satisfy a specific user need ( e.g. , generating images of a cat lying on the grass/ground/falling leaves).
Technically, we apply Principal Component Analysis (PCA) to obtain the desired concept subspaces from representative text embedding that correspond to a specific user need or requirement.
Projecting the text embedding of a given prompt into these low-dimensional subspaces enables efficient model editioning without retraining.
Intuitively, our proposed editioning paradigm enables a service provider to customize the base model into its “cat edition” (or other editions) that restricts image generation to cats, regardless of the user’s prompt ( e.g. , dogs, people, etc.).
This introduces a new dimension for product differentiation, targeted functionality, and pricing strategies, unlocking novel business models for text-to-image generators.
Extensive experimental results demonstrate the validity of our approach and its potential to enable a wide range of customized text-to-image model editions across various domains and applications.

## 1 Introduction

Recent advances in text-to-image models [, , , , , , ] have revolutionized visual content creation, enabling users to create highly realistic images from natural language descriptions.
However, as these models become more widely adopted, service providers face challenges in monetizing them and tailoring offerings to diverse customer needs. In the software industry, providers overcome this by offering product editions or versions tailored to specific user segments, e.g., Home Edition, Professional Edition, Enterprise Edition. In this work, we propose a novel task, namely, training-free editioning, and applies this strategy to text-to-image models.

Our core idea is creating model variations without retraining to cater to differing customer needs or offer distinct features, formulating editions as concept subspaces within the embedding space of the model’s text encoder. Our concept subspace encapsulates points satisfying a user requirement (i.e., concept), e.g., cat images with specific choices of backgrounds.
Technically, we apply Principal Component Analysis (PCA) to text embeddings corresponding to a given concept and retain principal components capturing key variations to obtain a low-dimensional subspace for that concept within the original embedding space.
Then, we achieve training-free editioning of text-to-image models by projecting the embeddings of input prompts into these subspaces.

Crucially, our approach allows service providers to efficiently customize the base model into targeted “editions” satisfying diverse customer needs without costly retraining. This leverages the pre-trained model’s capabilities while enabling fine-grained control over outputs. Providers can create tailored editions for different verticals, user types, or functionality tiers - e.g. a “cat edition” restricting outputs to cat images regardless of the input prompt (e.g., dogs, people). This unlocks innovative product strategies like freemium models with basic free editions versus feature-rich premium paid editions, enforcing content filters, specialty domains, or custom functionality per edition. Rather than offering an open-ended general tool, our paradigm shifts text-to-image models towards a customizable product portfolio optimized for commercial deployment. Service providers gain flexibility to create an offering tailored to their customer base, introducing novel business models beyond simply vending the base model. This empowers profitably serving diverse market needs while monetizing their AI assets through product differentiation and pricing opportunities better matched to consumer segments.
Extensive experiments validate our method’s ability to create purposeful model customizations across various domains and applications.
Our contributions include:

- •
We introduce a novel task called “training-free editioning” for text-to-image models, which aims to create customized variations or editions of a base model without expensive retraining.
- •
We propose a novel method to achieve training-free editioning by formulating different model editions as concept subspaces within the text embedding space of the base model, leveraging Principal Component Analysis (PCA) to obtain low-dimensional subspaces capturing desired concepts.
- •
Extensive experiments across various domains demonstrate the effectiveness of our approach in creating purposeful model customizations suited for different user groups and applications. We highlight the business potential of training-free editioning in enabling service providers to offer differentiated product editions, innovative pricing strategies, and tailored solutions optimized for commercial deployment.

Figure: Figure 1: Illustration of Text-to-Image Model Editioning. Our method can create variations (e.g., Boy Edition, Cat Edition) of a base text-to-image model without retraining, enabling them to cater to the diverse needs of different user groups or to offer distinct features and functionalities.
Refer to caption: x1.png

## 2 Related Work

Text-to-Image Synthesis.
Driven by the success of deep generative models, text-to-image synthesis has become a rapidly evolving field in computer vision and machine learning that aims to generate realistic images from textual descriptions.
One of the pioneering works in this field is AlignDRAW [], which introduced an attention-based approach that generates images by drawing a sequence of patches on the canvas based on an input caption.
While this method represented a promising step forward, the generated images often lacked coherence and failed to accurately reflect the input textual descriptions.
The advent of generative adversarial networks (GANs) ushered in a new era of text-to-image synthesis techniques. Text-conditional GANs [] were among the first to leverage the adversarial training framework for this task. Subsequently, methods like StackGAN [], AttnGAN [], and ControlGAN [] demonstrated improved performance by incorporating attention mechanisms and hierarchical architectures. Despite their notable achievements, these GAN-based approaches often struggled to maintain high consistency, resolution, and diversity in the generated images, falling short of meeting the demanding requirements of real-world applications.
A significant breakthrough in text-to-image synthesis emerged with the introduction of large-scale datasets and transformer-based models. OpenAI’s DALL-E [] pioneered the use of vast text-image pairs, enabling the generation of high-quality images from textual descriptions. Building upon this success, Parti [] further demonstrated the potential of scaling up data and models for improved text-to-image generation performance.
Nevertheless, thanks to their stable training and flexible conditioning (e.g., text, image, and other modalities), diffusion models [] have dominated the state-of-the-art solutions for text-to-image synthesis.

Diffusion Models.
Diffusion models are a class of deep generative models that have recently demonstrated remarkable performance in generating high-quality samples across various applications.
These models are parameterized Markov chains trained using variational inference to generate samples that match the data distribution after a finite number of iterations [, ]. Diffusion implicit models [], which are based on a class of non-Markovian diffusion processes, lead to the same training objective as traditional diffusion models, but can produce high-quality samples more efficiently.
A representative framework for training diffusion models in the latent space is Stable Diffusion, a scaled-up version of the Latent Diffusion Model (LDM) [].
Thanks to its flexibility allowing for multi-modal control signals (including text), Stable Diffusion has captivated the imagination of many users and dominated the field, especially in the open-source community.
For example, [] proposed a novel approach to create variations of a given “concept” by representing it with a single word embedding; Prompt-to-prompt [] focuses on manipulating the attention maps corresponding to the text embeddings for editing images in pre-trained text-conditioned diffusion models; Null-text inversion [] proposed performing DDIM inversion on the input image with related prompts into the latent space of a text-guided diffusion model, enabling intuitive text-based image editing.
These efforts, while effective, have focused primarily on extending the technical power and usability of the text-to-image (diffusion) model, whose business model is still immature.

To bridge this gap, we propose a novel task called training-free editioning for text-to-image models, which aims to create customized variations or editions of a base model without expensive retraining. This enables service providers to offer differentiated product editions, innovative pricing strategies, and tailored solutions optimized for commercial deployment.

## 3 Task Definition

We define the training-free editioning task for text-to-image models as follows:

###### Definition 1 (General) .

Given a trained general-purpose text-to-image model $M$, let $C=\{c_{1},c_{2},...,c_{n}\}$ be a list of $n$ concepts (textual) to be editioned on, $p$ be an input prompt to $M$, we denote the image synthesized by $M$ but editioned on $C$ as:

$$ $I=M(p\mid C),$ (1) $$

where $I$ is restricted to only contain concepts in $C$.

Despite its generality, Definition 1 lacks specifications for both the concept list $C$ and prompts $p$, posing challenges for both problem-solving and evaluation. As a first step in addressing this grand challenge, we propose a “Special” definition of the proposed task as follows:

###### Definition 2 (Special) .

Given a trained general-purpose text-to-image model $M$, let $p_{T}$ be an input prompt to $M$ following a template $T$ with one of the most basic linguistic topologies:

$$ $T=<\mathrm{subject}><\mathrm{verb}><\mathrm{preposition}><\mathrm{object}>,$ (2) $$

and let $C=\{c_{1},c_{2},...,c_{n}\}$ be a list of concepts for all elements in $p_{T}$ (e.g., “cat” for subject, “jumping” for verb, “grass” for object), we denote the corresponding image synthesized by $M$ but editioned on $C$ as:

$$ $I=M(p_{T}\mid C)$ (3) $$

where the subject/verb/object in $I$ belongs to $C$.

In this work, we focus on solving the problem outlined in Definition 3, which is an important first step towards solving the grand challenge and encompasses a large number of real-world use cases.

###### Definition 1 (General) .

###### Definition 2 (Special) .

### 3.1 Differences with Image Editing and Concept Erasing

Editioning vs. Editing.
Task-wise, text-to-image editing [, , , , , , , ] works at the aspect/image-level, which typically refers to the process of modifying or manipulating specific aspects of an existing image based on a text prompt or instructions while leaving irrelevant aspects of it unchanged, e.g., inpainting, outpainting, or style transfer.
In contrast, the proposed text-to-image editioning task works at the model-level, which aims to customize the behavior of the text-to-image model itself to cater to specific user needs or functionalities.
Methodology-wise, since editing works at the aspect/image-level, its key challenge is to disentangle the target aspect of an image that needs to be edited from the rest, e.g., by manipulating the attention maps of a given image [, , , , ].
On the contrary, our editioning task is performed at the model level, so its main challenge lies in controlling the model’s behavior, e.g., by manipulating the model’s text/image embedding space.

Editioning vs. Concept Erasing.
Task-wise, our editioning and concept erasing can be viewed as complementary tasks, where our editioning aims to retain concept(s) $c$ from a model and concept erasing aims to remove $c$ from the model.
Methodology-wise, existing concept erasing methods [, , , , , , , ] primarily focus on fine-tuning model weights.
In contrast, our approach does not involve any training and concentrates on manipulating the model’s text/image embedding space directly.
The choice of such distinct methodologies stems from the observation that $C$ typically constitutes a relatively small subset compared to the entire set of concepts learned by the model. Consequently, for concept erasing, removing $C$ can be achieved through a minor perturbation of the model weights. However, for our editioning task, retaining $C$ and dropping all other concepts would necessitate a significant modification, akin to retraining the entire model from scratch.

## 4 Method

Addressing the problem outlined in Definition 3, a straightforward idea is to modify $p_{T}$ by replacing its elements with those in $C$, i.e.:

$$ $p_{T}\mid C=p_{T(C)}$ (4) $$

where $T(C)=<{\rm subject}\in C><\mathrm{verb}\in C><\mathrm{preposition}\in C><
\mathrm{object}\in C>$.
However, this approach either requires prior knowledge of the template or requires complex syntax analysis to locate the replacement, and is thus not generalizable.
To address this issue, we propose a novel approach, namely Concept Subspace Projection, which shifts the focus from prompts $p_{T}$ to their embeddings and achieves $M(p_{T}\mid C)$ by projecting the embedding vector of $p_{T}$ to a concept subspace $S_{T(C)}$ defined by $T(C)$.
Specifically, let $\mathcal{E}$ be the text encoder of $M$ and $M(\cdot)=\hat{M}(\mathcal{E}(\cdot))$, $S=\mathbb{R}^{d}$ be the text (image) embedding space of $M$, we have:

$$ $I=M(p_{T}\mid C)=\hat{M}(\mathcal{E}(p_{T}\mid C))=\hat{M}(PR_{S_{T(C)}}( \mathcal{E}(p_{T})))$ (5) $$

where $PR_{x}(y)$ denotes the projection of $y$ on $x$, $S_{T(C)}=\mathbb{R}^{d_{T(C)}}\subset S$ denotes the concept subspace specified by the template $T$ and concepts $C$, $d_{T(C)}<d$.
In this way, we create the $\mathbf{C}$-edition of $M$ as the concept space $S_{T(C)}$.

Figure: Figure 2: Overview of our concept subspace creation (top) and projection (bottom).
Refer to caption: x2.png

### 4.1 CLIP-based Concept Subspace Projection

Recognizing that CLIP [] dominates the implementation of $\mathcal{E}$ in state-of-the-art text-to-image models [, , , , , , ], we follow this common practice and develop our method in the CLIP embedding space. Thanks to CLIP’s use of cosine similarity for comparing text and image embeddings, we hypothesize that (please see Sec. 5.3 for an empirical justification):

###### Conjecture 1 .

For CLIP encoders, the text embeddings in concept subspaces $S_{T(C)}$ corresponding to different $C$ are on a thin hypersphere shell centered at the origin.

Concept Subspace Creation.
Based on Conjecture 1, the concept subspace $S_{T(C)}$ accommodating the hypersphere shell can be characterized by a set of (orthogonal) vectors radiating from the origin.
To obtain such vectors, we propose applying Principal Component Analysis (PCA) to a substantial sample of $\mathcal{E}(p_{T(C)})$ embeddings $D_{C}=[\mathcal{E}(p^{1}_{T(C)}),\mathcal{E}(p^{2}_{T(C)}),...,\mathcal{E}(p^{
m}_{T(C)})]$:

$$ $\mathbf{V}_{C}=\mathrm{PCA}(D_{C})$ (6) $$

where $\mathbf{V}_{C}$ denotes the principal axes ranked by descending principal values. Then, we define:

$$ $\hat{\mathbf{V}}_{C}(k)=\mathbf{V}_{C}[0:k]$ (7) $$

as the basis of subspace $S_{T(C)}$, where $k$ is selected according to a 95% threshold of explained variance.
Please see Sec. 5.4 for more details.

Magnitude-compensated Projection.
With $\hat{\mathbf{V}}_{c}(k)$, we define the projection function $PR$ as:

$$ $PR_{S_{T(C)}}(\mathcal{E}(p_{T}))=\eta\cdot\hat{\mathbf{V}}_{c}(k)\cdot\hat{ \mathbf{V}}_{c}(k)^{T}\cdot\mathcal{E}(p_{T})$ (8) $$

where $\eta=\frac{||\mathcal{E}(p_{T})||}{||\hat{\mathbf{V}}_{c}(k)\cdot\hat{\mathbf{
V}}_{c}(k)^{T}\cdot\mathcal{E}(p_{T})||}$ is a parameter to compensate for the loss of magnitude during the projection.
Note that we omit the centering step for simplicity, since the PCA subspace is also approximately centered at the origin (Conjecture 1).

###### Conjecture 1 .

##### Efficient Computation.

Figure: Figure 3: 13k components yield a cumulative explained variance ratio of 99.9+%.
Refer to caption: extracted/5623322/Explained_Variance_Ratio/Explained_Variance_Ratio.png

Guided by the classic manifold hypothesis [] that assumes the existence of low-dimensional representations of high-dimensional data, we apply PCA to a substantial random sample of CLIP text embeddings $D_{all}$ to reduce the dimensionality of the original CLIP embedding space from $77\times 768=59,136$ to $13,000$.
This “compression” significantly improves computational efficiency by roughly $(59,136/13,000)^{2}\approx 20.7$ for the computation of covariance matrix in PCA (bottleneck), without sacrificing the performance (Fig. 3).
Thus, unless specified otherwise, we employ the 13,000-dimensional reduced space as the CLIP text embedding space in our experiments.

## 5 Experiments

### 5.1 Experimental Setup

As mentioned above, our method consists of two steps: i) performing a low-loss dimensionality reduction to obtain an “efficient” CLIP embedding space of 13,000 dimensions; ii) creating concept spaces and projecting the embeddings of input prompts to them using the method proposed in Sec. 4.
To implement them, we created the following datasets:

CLIP Dimensionality Reduction Dataset ($D_{all}$). CoCo 2017 Dataset [] contains thousands of image and caption pairs, where each caption briefly describes the object, action, and subject of the image, e.g., “a cat laying on top of a counter looking down”. We randomly selected a subset of 160,000 captions from it and embedded them with CLIP to create the dataset $D_{all}$ for the dimensionality reduction in creating the $13,000$-dimension “efficient” CLIP embedding space.

Concept Datasets. First, we construct a word list $WL$ for each element of the template $T=<\mathrm{subject}><\mathrm{verb}><\mathrm{preposition}><\mathrm{object}>$ (Eq. 2) and create a base dataset $D_{\rm all_concept}$ containing all combinations of prompts created using $T$ and $WL$.
Without loss of generality, we focus on the different $<\mathrm{subject}>$ in our main experiments as they represent an important type of text-to-image model edition.
Thus, we create the concept datasets for each $<\mathrm{subject}>$ by filter out all the prompts in $D_{\rm all_concept}$ that do not contain the $<\mathrm{subject}>$ (e.g., $D_{\rm cat}$, $D_{\rm dog}$).
Please see Appendix A for more details.

Concept Subspaces Creation and Evaluation. We follow the method detailed in Sec. 4.1 to create our concept subspaces (e.g., $S_{\rm cat}$, $S_{\rm dog}$) using their corresponding concept datasets (e.g., $D_{\rm cat}$, $D_{\rm dog}$), respectively. In addition, given a concept subspace $S_{<\mathrm{subject}>}$, we construct its evaluation dataset by sampling 1,000 random samples for each of the 8 categories in the complement set $\hat{D}_{<\mathrm{subject}>}=D_{\rm all_concept}\backslash D_{<\mathrm{
subject}>}$, totaling 8,000 samples.

Evaluation Metrics.
We use i) the CLIP score [] to measure the consistency between an image generated by our method and its corresponding descriptions (before and after our content subspace projection); ii) Frechet Inception Distance (FID) [] and Inception Score (IS) to measure the image synthesis ability of the base model and its editions created by our method.

**Table 1: CLIP score (softmax probability) of the images generated by our concept subspace projection, and their corresponding “ground truth” prompts (i.e., those accurately describing the image content).**
| Concept Subspace | Categories of Prompts in the Evaluation Datasets |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dog | Cat | Tiger | Car | Bus | Truck | Boy | Girl | Man |  |
| Dog Ed. ($S_{\rm dog}$) |  | 0.9906$\pm$0.0550 | 0.9818$\pm$0.0830 | 0.9778$\pm$0.0959 | 0.9806$\pm$0.0899 | 0.9742$\pm$0.1075 | 0.9490$\pm$0.1172 | 0.9672$\pm$0.1054 | 0.9582$\pm$0.1539 |
| Cat Ed. ($S_{\rm cat}$) | 0.9938$\pm$0.0413 |  | 0.9302$\pm$0.1122 | 0.9786$\pm$0.1030 | 0.9859$\pm$0.0710 | 0.9783$\pm$0.1049 | 0.9463$\pm$0.1382 | 0.9632$\pm$0.1242 | 0.9624$\pm$0.1438 |
| Tiger Ed. ($S_{\rm tiger}$) | 0.9967$\pm$0.0356 | 0.9935$\pm$0.0278 |  | 0.9576$\pm$0.1633 | 0.9315$\pm$0.1869 | 0.8927$\pm$0.2465 | 0.9650$\pm$0.1487 | 0.9799$\pm$0.1018 | 0.9423$\pm$0.2051 |
| Car Ed. ($S_{\rm car}$) | 0.9580$\pm$0.1074 | 0.9435$\pm$0.1292 | 0.9445$\pm$0.1414 |  | 0.8950$\pm$0.2514 | 0.9551$\pm$0.1362 | 0.9210$\pm$0.1694 | 0.9145$\pm$0.1792 | 0.8740$\pm$0.2180 |
| Bus Ed. ($S_{\rm bus}$) | 0.8972$\pm$0.2474 | 0.9149$\pm$0.2206 | 0.9479$\pm$0.1676 | 0.9211$\pm$0.2243 |  | 0.9495$\pm$0.1728 | 0.9240$\pm$0.2232 | 0.8937$\pm$0.2603 | 0.8764$\pm$0.2894 |
| Truck Ed. ($S_{\rm truck}$) | 0.9471$\pm$0.1626 | 0.9265$\pm$0.1840 | 0.9764$\pm$0.0905 | 0.9155$\pm$0.2216 | 0.9830$\pm$0.0736 |  | 0.9573$\pm$0.1569 | 0.9121$\pm$0.2195 | 0.8997$\pm$0.2324 |
| Boy Ed. ($S_{\rm boy}$) | 0.9891$\pm$0.0526 | 0.9887$\pm$0.0652 | 0.9855$\pm$0.0569 | 0.9732$\pm$0.0963 | 0.9938$\pm$0.0448 | 0.9879$\pm$0.0606 |  | 0.9676$\pm$0.0919 | 0.9340$\pm$0.1320 |
| Girl Ed. ($S_{\rm girl}$) | 0.9920$\pm$0.0334 | 0.9907$\pm$0.0548 | 0.9905$\pm$0.0454 | 0.9612$\pm$0.1253 | 0.9884$\pm$0.0509 | 0.9804$\pm$0.0764 | 0.9261$\pm$0.1179 |  | 0.9536$\pm$0.1236 |
| Man Ed. ($S_{\rm man}$) | 0.9788$\pm$0.0665 | 0.9817$\pm$0.0659 | 0.9668$\pm$0.0861 | 0.9403$\pm$0.1545 | 0.9845$\pm$0.0677 | 0.9624$\pm$0.1134 | 0.8421$\pm$0.1923 | 0.9524$\pm$0.1411 |  |

**Table 2: Evaluating the image synthesis capability of our concept space projection method using FID and IS scores. Ours: for each concept subspace, we take its evaluation dataset and generate their corresponding 8,000 images using the proposed concept space projection; SD: for each concept subspace, we replace the <object> of the prompts used in “Ours” with the concept of the subspace and generate 8,000 images using Stable Diffusion v1.4 []). SD’: different sets of 8,000 images generated in the same way as “SD”.**
|  |  | Dog | Cat | Tiger | Car | Bus | Truck | Boy | Girl | Man | Mean |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| FID | Ours vs. SD | 13.996 | 17.277 | 33.197 | 13.014 | 33.404 | 22.555 | 13.750 | 13.795 | 15.292 | 19.587 |
| SD vs. SD’ | 6.723 | 6.143 | 2.390 | 6.239 | 3.093 | 3.887 | 9.977 | 10.035 | 11.081 | 6.619 |  |
| IS | Ours | 10.081 | 3.964 | 2.678 | 5.693 | 4.030 | 3.774 | 9.361 | 9.421 | 11.125 | 6.680 |
| SD | 8.551 | 2.531 | 1.155 | 4.357 | 2.209 | 1.940 | 8.707 | 8.897 | 10.507 | 5.428 |  |

### 5.2 Effectiveness of Concept Subspace Projection for Text-to-Image Model Editioning

#### 5.2.1 Quantitative Results

Editioning Accuracy.
We use the CLIP score (probability) [] to measure the editioning accuracy of our method, with 0 indicating low accuracy and 1 indicating high accuracy. Specifically, given a concept space $S$ (e.g., cat edition $S_{\rm cat}$) created using $D$ (e.g., $D_{\rm cat}$) defined in Sec. 5.1, for each input prompt $\hat{p}$ in its corresponding evaluation dataset $\hat{D}$ (e.g., $\hat{D}_{\rm cat}$, all prompts without “cat”), we compute the softmax probability of:
i) the CLIP score between the images $I$ generated using the projected prompts and their corresponding “ground truth” prompts $p$, i.e., the $p$ in $D$ that is equivalent to replacing the corresponding concept $c$ in $\hat{p}$ with that of $D$ (e.g., “dog” to “cat” in $p$);
ii) the CLIP score between $I$ and $p$.
Since the sum of the two probabilities is 1, we report the former in Table 1, which demonstrates that our concept subspace projection accurately restricts the generation to the concept of $S$ (all scores are high).

**Table 3: Cosine similarities between the input prompt, its projected version, and its “replaced” version. The “replaced” version refers to the text embedding of the prompt created by replacing the target component in the input prompt (e.g., “dog”) with that of the concept subspace (e.g., “cat”) being projected onto. The input prompts used are from the evaluation dataset of the corresponding subspace.**
|  | $d({\rm input},{\rm replace})$ | $d({\rm project},{\rm replace})$ |  | $d({\rm input},{\rm replace})$ | $d({\rm project},{\rm replace})$ |
| --- | --- | --- | --- | --- | --- |
| Dog Ed. ($S_{\rm dog}$) | 0.212$\pm$0.050 | 0.059$\pm$0.018 | Truck Ed. ($S_{\rm truck}$) | 0.250$\pm$0.064 | 0.102$\pm$0.031 |
| Cat Ed. ($S_{\rm cat}$) | 0.215$\pm$0.058 | 0.067$\pm$0.022 | Boy Ed. ($S_{\rm boy}$) | 0.206$\pm$0.070 | 0.063$\pm$0.022 |
| Tiger Ed. ($S_{\rm tiger}$) | 0.253$\pm$0.063 | 0.085$\pm$0.025 | Girl Ed. ($S_{\rm girl}$) | 0.221$\pm$0.067 | 0.064$\pm$0.024 |
| Bus Ed. ($S_{\rm bus}$) | 0.275$\pm$0.062 | 0.108$\pm$0.029 | Man Ed. ($S_{\rm man}$) | 0.198$\pm$0.061 | 0.062$\pm$0.021 |
| Car Ed. ($S_{\rm car}$) | 0.217$\pm$0.052 | 0.076$\pm$0.025 | Mean | 0.227 | 0.076 |

Image Synthesis Capability.
We use the FID and IS scores to measure the image synthesis capability, with reference to those of the base model, i.e., Stable Diffusion (SD) v1.4 [].
As Table 2 shows, the FID scores of our method are worse than those of SD but our IS scores are higher, indicating that our method generates similarly high quality but less diverse images than SD.

Similarity between Text Embeddings.
To further characterize our content subspace projection, we compute the cosine similarities between the input prompt, its projected version, and its “replaced” version, where the “replaced” version refers to the text embedding of the prompt created by replacing the target component in the input prompt (e.g., “dog”) with that of the concept subspace (e.g., “cat”) being projected onto.
As Table 3 shows, the projected embeddings are very close to the “replaced” ones, indicating that our projection is successful.

#### 5.2.2 Qualitative Results

Editioning Accuracy.
As Fig. 4 shows, we achieved a high editioning accuracy as the target concept (i.e., $<\rm subject>$) of the input prompt is restricted to the concept subspace while other concepts (i.e., $<\rm verb>$, $<\rm object>$) remain unchanged.

Figure: Figure 4: Images generated by different prompts when using different editions of the Stable Diffusion v1.4 model. The input prompts are shown at the top.
Refer to caption: x3.png

Image Synthesis Capability.
As Fig. 5 shows, the images generated using our concept subspace projection are of similarly high quality and diversity to those generated by the base model directly.

Figure: Figure 5: Different images generated by the same prompt when using different editions of the Stable Diffusion v1.4 model. The input prompts are shown at the bottom.
Refer to caption: x4.png

### 5.3 Properties of CLIP Concept Subspaces

Distance to Origin.
As Fig. 6 shows, we plot the histogram of the distances of text embeddings to the origin for each concept dataset and the Coco 2017 dataset.
It can be observed that for all datasets, the distances are around 250 with small standard deviations, which justifies our Conjecture 1.

Figure: Figure 6: Distances of text embeddings to the origin. We randomly selected 2,000 prompts from each concept dataset and the Coco 2017 dataset to calculate the distances of samples to the origin. The mean and standard deviation values of the distances are shown in the legend.
Refer to caption: extracted/5623322/distance_of_each_sample.png

Semantic Directions in Concept Subspace.
As Fig. 7 shows, we observed that the principal components of each concept subspace also have semantic meanings. In addition, the content of the image remains restricted to its corresponding conceptual subspace (edition).

Figure: Figure 7: Images generated by moving text embeddings (the input prompt is shown in the middle) along the directions of principal components (PC). Row 1: PC #7; Row 2: PC #14; Row 3: PC #0.
Refer to caption: x7.png

Concept Subspace Interpolation.
As Fig. 8 shows, our concept subspace also allows for linear interpolation between projected text embeddings.

Figure: Figure 8: Linear interpolation between projected text embeddings. The input prompts are shown at the bottom. The words in red denote the <subject> to be restricted to the edition given on the left.
Refer to caption: x9.png

**Table 4: Choice of $k$ (Eq. 7) by 95% explained variance ratio for each concept subspace.**
|  | Dog | Cat | Tiger | Car | Bus | Truck | Boy | Girl | Man |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| # of Principal Components | 30 | 29 | 27 | 29 | 30 | 30 | 33 | 33 | 34 |

### 5.4 Choice of k for Each Concept Subspace

As Table 4 shows, empirically, we choose $k$ (Eq. 7) for each concept subspace by the threshold of 95% explained variance ratio of that subspace.

## 6 Conclusion

Inspired by software editioning, we propose training-free “editioning” of text-to-image models by identifying concept subspaces within the latent space of their text encoders (e.g., CLIP). These subspaces, obtained via applying Principal Component Analysis (PCA) on representative text embeddings, correspond to specific domains or attributes like “generating images of a cat lying on the grass/ground/falling leaves”. Projecting the text embedding of a given prompt into these low-dimensional subspaces enables efficient model customization without retraining.
This unlocks novel business models, such as offering restricted “cat editions” that only generate cat images regardless of objects in input prompts, enabling new product differentiation and pricing strategies.

## Appendix A Details of Concept Datasets

As Table 5 shows, we construct a word list $WL$ for each element of the template $T=<\mathrm{subject}><\mathrm{verb}><\mathrm{preposition}><\mathrm{object}>$ (Eq. 2) and create a base dataset $D_{\rm all_concept}$ containing all combinations of prompts created using $T$ and $WL$, resulting in a total of $9\times 21\times 8\times 21=31,752$ prompts.
Without loss of generality, we focus on the different $<\mathrm{subject}>$ in our main experiments as they represent an important type of text-to-image model edition.
Thus, we create the concept datasets for each $<\mathrm{subject}>$ by filtering out all the prompts in $D_{\rm all_concept}$ that do not contain the $<\mathrm{subject}>$ (e.g., $D_{\rm cat}$, $D_{\rm dog}$), resulting in $21\times 8\times 21=3,528$ prompts per subspace.

**Table 5: Word list for each element of the template $T=<\mathrm{subject}><\mathrm{verb}><\mathrm{preposition}><\mathrm{object}>$ (Eq. 2).**
| $\mathrm{subject}$ | $\mathrm{verb}$ | $\mathrm{preposition}$ | $\mathrm{object}$ |
| --- | --- | --- | --- |
| cat | sleeping | on | ground |
| tiger | exploring | with | sky |
| dog | moving | in | river |
| car | lounging | over | bed |
| bus | lying | through | chair |
| truck | gazing | out | desk |
| boy | chasing | besides | grass |
| girl | stretching | under | seat |
| man | standing |  | couch |
|  | stalking |  | backyard |
|  | leaping |  | garden |
|  | talking |  | house |
|  | running |  | leaves |
|  | jumping |  | road |
|  | napping |  | station |
|  | staying |  | path |
|  | sitting |  | stop |
|  | smiling |  | street |
|  | racing |  | yard |
|  | stopping |  | field |
|  | waiting |  | way |

## Appendix B Results on Stable Diffusion v1.5

As shown in Table 6 and Fig. 9, our method also generalizes to Stable Diffusion v1.5 and achieves similarly high CLIP scores and editioning accuracy.

**Table 6: CLIP score (softmax probability) of the images generated by our concept subspace projection, and their corresponding “ground truth” prompts (i.e., those accurately describing the image content).**
| Concept Subspace | Categories of Prompts in the Evaluation Datasets |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dog | Cat | Tiger | Car | Bus | Truck | Boy | Girl | Man |  |
| Dog Ed. ($S_{\rm dog}$) |  | 0.9923$\pm$0.0523 | 0.9864$\pm$0.0804 | 0.9778$\pm$0.1054 | 0.9852$\pm$0.0750 | 0.9634$\pm$0.1451 | 0.9605$\pm$0.0956 | 0.9739$\pm$0.1004 | 0.9816$\pm$0.0762 |
| Cat Ed. ($S_{\rm cat}$) | 0.9891$\pm$0.0694 |  | 0.9370$\pm$0.1177 | 0.9781$\pm$0.1102 | 0.9813$\pm$0.0878 | 0.9600$\pm$0.1525 | 0.9540$\pm$0.1292 | 0.9609$\pm$0.1460 | 0.9557$\pm$0.1719 |
| Tiger Ed. ($S_{\rm tiger}$) | 0.9907$\pm$0.0645 | 0.9928$\pm$0.0293 |  | 0.9438$\pm$0.1837 | 0.9562$\pm$0.1450 | 0.9262$\pm$0.1990 | 0.9675$\pm$0.1471 | 0.9815$\pm$0.0945 | 0.9352$\pm$0.2149 |
| Car Ed. ($S_{\rm car}$) | 0.9714$\pm$0.0853 | 0.9571$\pm$0.1075 | 0.9711$\pm$0.0747 |  | 0.9605$\pm$0.1437 | 0.9741$\pm$0.0813 | 0.9558$\pm$0.1119 | 0.9322$\pm$0.1540 | 0.8941$\pm$0.1876 |
| Bus Ed. ($S_{\rm bus}$) | 0.9748$\pm$0.1147 | 0.9813$\pm$0.0873 | 0.9895$\pm$0.0571 | 0.9565$\pm$0.1628 |  | 0.9806$\pm$0.0989 | 0.9897$\pm$0.0768 | 0.9737$\pm$0.1312 | 0.9701$\pm$0.1397 |
| Truck Ed. ($S_{\rm truck}$) | 0.9883$\pm$0.0660 | 0.9743$\pm$0.1046 | 0.9862$\pm$0.0852 | 0.9666$\pm$0.1227 | 0.9962$\pm$0.0202 |  | 0.9896$\pm$0.0498 | 0.9863$\pm$0.0788 | 0.9759$\pm$0.0692 |
| Boy Ed. ($S_{\rm boy}$) | 0.9932$\pm$0.0322 | 0.9924$\pm$0.0450 | 0.9916$\pm$0.0499 | 0.9684$\pm$0.1272 | 0.9923$\pm$0.0563 | 0.9821$\pm$0.0852 |  | 0.9730$\pm$0.0836 | 0.9571$\pm$0.1059 |
| Girl Ed. ($S_{\rm girl}$) | 0.9900$\pm$0.0530 | 0.9934$\pm$0.0243 | 0.9919$\pm$0.0465 | 0.9686$\pm$0.1093 | 0.9874$\pm$0.0643 | 0.9824$\pm$0.0781 | 0.9330$\pm$0.1145 |  | 0.9688$\pm$0.0974 |
| Man Ed. ($S_{\rm man}$) | 0.9796$\pm$0.0777 | 0.9890$\pm$0.0395 | 0.9806$\pm$0.0647 | 0.9334$\pm$0.1783 | 0.9911$\pm$0.0390 | 0.9668$\pm$0.1040 | 0.8698$\pm$0.1644 | 0.9700$\pm$0.0922 |  |

Figure: Figure 9: Images generated by different prompts when using different editions of the Stable Diffusion v1.5 model. The input prompts are shown at the top.
Refer to caption: x12.png

## Appendix C Concept Subspaces of < verb > expectation verb <{\rm verb}> < roman_verb > and < object > expectation object <{\rm object}> < roman_object >

Following a similar experimental setup used for $<{\rm subject}>$ in the main paper, we show that the proposed method can also be applied to $<{\rm verb}>$ and $<{\rm object}>$ of the template $T=<\mathrm{subject}><\mathrm{verb}><\mathrm{preposition}><\mathrm{object}>$ (Eq. 2).
As shown in Table 9(b), Fig. 10, Fig. 11, our method can also accurately restrict the generation to the concept subspace.

**Table 7: CLIP score (softmax probability) of the images generated by our concept subspace projection, and their corresponding “ground truth” prompts (i.e., those accurately describing the image content) for (a) $<{\rm verb}>$ and (b) $<{\rm object}>$ of the template $T=<\mathrm{subject}><\mathrm{verb}><\mathrm{preposition}><\mathrm{object}>$ (Eq. 2)**
| Concept Subspace | Verb |  |  |
| --- | --- | --- | --- |
| Sitting | Sleeping | Racing |  |
| Sitting Ed. ($S_{\rm sitting}$) |  | 0.8085$\pm$0.2048 | 0.8060$\pm$0.2373 |
| Sleeping Ed. ($S_{\rm sleeping}$) | 0.8011$\pm$0.2544 |  | 0.8051$\pm$0.2743 |
| Racing Ed. ($S_{\rm racing}$) | 0.8167$\pm$0.2685 | 0.9001$\pm$0.1980 |  |

Figure: Figure 10: Concept Subspaces of $<{\rm verb}>$. Images generated by different prompts when using different editions of the Stable Diffusion v1.4 model. The input prompts are shown at the top. The left clarifies the different editions of the object and the base model.
Refer to caption: x13.png

Figure: Figure 11: Concept Subspaces of $<{\rm object}>$. Images generated by different prompts when using different editions of the Stable Diffusion v1.4 model. The input prompts are shown at the top. The left clarifies the different editions of the object and the base model.
Refer to caption: x14.png

## Appendix D Effectiveness of Our Magnitude-compensated Projection

Distances of Text Embeddings to the Origin after Naive Projection. To demonstrate the necessity of our magnitude-compensated projection (Eq. 8), we show that the distances indeed reduce after naive projections (Fig. 12).

Qualitative Comparison. As Fig. 13 shows, without our magnitude-compensated projection (i.e., naive projection), the generated images suffer from severe distortions, which further demonstrates the effectiveness of our magnitude-compensated projection.

Figure: Figure 12: Distances of text embeddings to the origin after naive projection. We randomly selected 2,000 prompts from the evaluation dataset of a given concept space $S$ and naively projected them to $S$. The mean and standard deviation values of the distances are shown in the legend.
Refer to caption: extracted/5623322/projection_distance_of_each_sample.png

Figure: Figure 13: Comparison of images generated using naive projection and our magnitude-compensated projection. All images are generated with editions of the Stable Diffusion v1.4 model. The input prompts are shown at the top. The editions are shown at the bottom.
Refer to caption: x15.png

## Appendix E Limitations

Our work is a first step toward the new task of “Training-free Editing of Text-to-image Models”. As such, it is constrained by the templates used, a limited list of words, and other factors. Nonetheless, we believe that our approach is a solid step forward and will inspire the community for subsequent innovations.

## Appendix F Computational Resources

All experiments in our work are conducted on a workstation with an 12th-gen Intel Core i7-12700 CPU, an Nvidia RTX 4090 24G GPU, 64GB memory and a 1TB hard disk.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: - The motivation of this paper needs to be better explained.
- The proposed method's technical novelty could be improved.
- Editing the prompts with LLMs would be interesting to investigate.
soundness: 2
strengths: - This paper aims to introduce a new type of task that ignores subjects in the text prompts while only editing a predefined concept (e.g., cat).

- This paper analyzes the PCA space in CLIP text encoder embedding and achieves image editioning.

- Several experiments and interesting observations are made for this new task and the proposed method.
summary: This paper proposes a new task named image editioning that only allows editing a specific concept without training the text-to-image model. To achieve this, this paper projects the CLIP text embedding with PCA into concept subspaces and uses the principal axes corresponding to the concept for editing. The resulting system could only generate images of the desired concept regardless of subjects in input prompts. Experiments demonstrate that the proposed method can achieve the proper results on this new proposed task.
weaknesses: - The image editioning setup is a bit weird to me. If I would like to only generate specific objects, why not just replace the original object's name with the desired concept? An LLM could easily be used for that purpose. The practical value of the proposed method should be better explained. 

- Different editions require the same computations, and I do not see why different editions should charge differently as illustrated in Figure 1. I have a hard time figuring out why this task could stimulate new business models. 

- What if the prompt contains multiple objects? Can this method select the correct one? Could the proposed method be extended to edit multiple objects/concepts? These points need some discussion.

- Many related works have been done for editing the styleGAN space for image editing, these methods should be cited and discussed.

[a]  GANSpace: Discovering Interpretable GAN Controls.
[b] A Compact and Semantic Latent Space for Disentangled and Controllable Image Editing.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: a.The proposed method is simple but effective. The author propose a novel task for offering different editions or versions of a product tailored to specific user groups or use cases, which may be useful in the software industry’s practice.

b.The author conducts extensive experiments and proves its effectiveness.
summary: This paper proposes a training-free image editioning method aimed for providing controllable image generation for specific user groups or uses cases. They apply PCA to obtain the desired concept subspaces and projects the text embedding of a given prompt into these desired concept low-dimensional subspaces enables efficient model editioning. Extensive experiments demonstrate the effectiveness of the proposed methods.
weaknesses: a. The proposed method, while eliminating the need for model retraining, still requires the creation of a desired concept dataset in advance, which somewhat restricts its applications. 

b. The presentation of method part is not very clear, especially Figure 3 and I didn’t find its citation in the main text.

c.What if I need some concepts that are rare or even not included in the pre-prepared dataset? 

d.What if  I want to generate images with multiple concepts rather than a single concept?

e. The pretrained model used in this paper is sd 1.4, what about its effectiveness on other diffusion models?

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Q1. How will the number of prompts in C influence the final performance?
Q2. How could the proposed method deal with multiple concepts? Besides subjects, how does it work on attributes?
soundness: 3
strengths: Overall, the reviewer appreciates the efforts in defining a new task for T2I applications, which genuinely will shed some new light on the research field. Despite much research that has tried to generate/find desired textual prompts for different goals, this is the first time the reviewer has considered controlling the output of T2I models by projecting the textual prompts. The proposed method, though somewhat straightforward, is reasonable. The discussions on Sec.3.1 are also welcomed.
summary: In this study, the authors propose a new task, ''training-free editioning'', for business T2I services. The goal of the new task is to force the base T2I model to only generate images from a predefined concept set. To avoid retraining the base model, the authors project the input textual prompts to the given concepts. Experiments show the method would be a valid solution for the proposed new task.
weaknesses: 1. Limited scope. Though the authors have tried their best to show the potential applications for the newly proposed task, the reviewer is not convinced that the proposed solution will be widely used in the T2I community. In what situation could we need to force a T2I service to generate only "cat" images while "must not" other concepts? For concept erasing or model aligning, the goal is clear and easy to achieve by listing unwanted concepts. Then, most of the common concepts would work as usual. In the proposed editioning setting, the outputs can only be one concept (which could also be an unsafe concept if the service wants).
2. The proposed PCA-based method is niche. When reading the definition in Section 3, the reviewer thought the concepts in C could be different objects or attributes. However, as shown in the experiments, one edition of T2I model could only support one kind of subject. Related to the first issue, such a setting is also nearly impractical. 
3. Technical writing should be given more attention. The reviewer found it hard to locate the manuscript in related works. The new task confused the reviewer when reading most parts of the introduction. Figure 1 is not helpful. The second paragraph comes in a strange way. The authors are suggested to re-phase the introduction to clearly show the context of this manuscript.
4. The reviewer is surprised to find no user study evaluation is provided in the manuscript.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. Please provide more results with better pre-trained models, like SDXL.
2. What’s the advantage of PAC in this paper compared with LLMs for text alignment. 
3. Provides more research papers which are related to the prompt processing and are published in 2024.
soundness: 1
strengths: This paper attempts to replace the concept with the PCA model in the subspace of the label, and implements it as training-free editing.
summary: This paper focus on the "training-free editioning" task based on a pre-trained text-to-image model. To achiever the objective, the paper proposes some aligned concepts for the input prompts, and apply the PCA method to obtain the desired subspaces where the input and based concepts can be consist with each other. The paper announced that they introduce the novel task.
weaknesses: 1. The task proposed in this paper is one of the most basic tasks of text-to-image task. 
2. A lot of works have done to align the subjects during the generating process. 
3. There is no comparative experiments with other outstanding models, like SDXL, FLUX 1.0. 
4. The related work part just list some older papers and there is  insufficient relevance to the question studied in this paper. 
5. In the method part, There is only a simple introduction to the PCA method in the 4.1 CLIP-BASED CONCEPT SUBSPACE PROJECTION. No more subject follow it. 
6. In the experiment part, the 5.2.2 and 5.3 just list the general words without any in-depth analysis.

Meta-review: This paper aims to study a new task which only allows editing a specific concept without training the text-to-image model. The authors propose an image editioning method to provide controllable image generation for specific user groups or uses cases. Experiments are performed to evaluate the effectiveness of the proposed method. Four reviewers raised concerns about the task setup, related work, novelty and presentation and they gave negative rating scores. The authors did not provide responses during the rebuttal period. Based on the above considerations, I do not recommend to accept this manuscript.

Decision: Reject