Wayward Concepts In Multimodal Models
Authors: Brandon Trabucco
Max A Gurinas
Kyle Doherty
Russ Salakhutdinov
Venue: ICLR 2025 Poster (2025)
Keywords: Deep Learning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Large multimodal models such as Stable Diffusion can generate, detect, and classify new visual concepts after optimizing just the prompt. How are prompt embeddings for visual concepts found by prompt tuning methods different from typical discrete prompts? We conduct a large-scale analysis on three state-of-the-art models in text-to-image generation, open-set object detection, and zero-shot classification, and find that prompts optimized to represent new visual concepts are akin to an adversarial attack on the text encoder. Across 4,800 new embeddings trained for 40 diverse visual concepts on four standard datasets, we find perturbations within an $\epsilon$-ball to any prompt that reprogram models to generate, detect, and classify arbitrary subjects. These perturbations target the final-layers in text encoders, and steer pooling tokens towards the subject. We explore the transferability of these prompts, and find that perturbations reprogramming multimodal models are initialization-specific, and model-specific. Code for reproducing our work is available at the following site: https://wayward-concepts.github.io.

Published as a conference paper at ICLR 2025
WAYWARD CONCEPTS IN MULTIMODAL MODELS
Brandon Trabucco', Max Gurinas", Kyle Doherty Ruslan Salakhutdinov 'Camegie Mellon University, UUniversity Of Chicago Laboratory Schools, "MPG Ranch brandon@btrabucco. com, rsalakhude cs. cmu. edu
ABSTRACT
Large multimodal models such as Stable Diffusion can generate, detect, and clas- sify new visual concepts after optimizing just the prompt. How are prompt em- beddings for visual concepts found by prompt tuning methods different from typ- ical discrete prompts? We conduct large-scale analysis on three state-of-the-arr models in text-to-image generation, open-set object detection, and zero-shot clas- sification, and find that prompts optimized to represent new visual concepts are akin to adversarial attack the text encoder. Across 4,800 new embeddings trained for 40 diverse visual concepts on four standard datasets, we find pertur- bations within e-ball any prompt that reprogram models generate, detect, and classify arbitrary subjects. These perturbations target the final-layers in text encoders, and steer pooling tokens towards the subject. We explore the transfer- ability of these prompts, and find that perturbations reprogramming multimodal models are inittalization-specffic and model-specific. Code for reproducing our work is available at the following site: wayward-concepps.gu github..
INTRODUCTION
Fine-tuning prompts is widely successful technique for adapting large pretrained models to new tasks from limited data (Lester et al.] [2021] Li & Liang] [021 Shhin et al. 2020 Gal et al.) 2023). In language modelling, these prompts can efficiently teach pretrained language models specialized tasks, such as reading tables (Li & Liang] 2021). In text-to-imant image generation, they can embed subjects with unique, often bard-to-describe appearances into the generations of diffusion model (Gal et al., 2023 Ruiz et al. 2023). Large multimodal models, such as Stable Diffusion (Rombach al. 2022). OWL-v2 (Minderer al., 2022), and CLIP (Radford et al. 2021 Cherti et al., [023] Fang et al. 2023), can generate, detect, and classify diverse visual concepts not present in their training data after fine-tuning embeddings representing that concept in their prompt (Gal et al. 2023 Trabucco al. 2023). How do soft prompts obtained via prompt tuning methods that encode specific visual concepts (i.e. black dog) differ from rypical discrete prompts? There is a popular hypothesis in multimodal machine learning that text-based models have emergent representations for visual content (Liu et al. [203a] Koh et al. 2023 Huh et al.) 2024 Lu et al.] (022), despite training purely on text. This investigation aims determine if visual prompt tuning methods find emergent embeddings akin to the existing discrete prompts, or if they find something more subtle.
We conduct large-scale study on three state-of-the-art models in text-to-image generation, open- set object detection, and zero-shot classification. We optimize 4,800 new embeddings for Stable Diffusion (Rombach et al. (2022), ,, OWL-v2 (Minderer et al. 2022). and CLIP (Radford al. 202T Cherti et al., 2023) Fang et al. 2023) to generate. detect, and classify 40 diverse visual concepts across four standard datasets. For all tested models and datasets, visual prompt tuning finds solutions within an e-ball to unrelated discrete prompts that reprogram models to generate, detect, and classify arbitrary subjects. We refer to this redundancy of solutions as fracturing of the embedding space. Fractured solutions found by visual prompt tuning have noteworthy properties: first, solutions robust to their location in the embedding space, and second, they target specific layers in the model.
Across all models and tasks, fractured solutions anchored to unrelated discrete prompts perform comparably to solutions anchored to related discrete prompts. In both cases, fractured solutions target the final layers of text encoders to steer text encoder representations away from the original discrete prompt, and towards the desired subject. We explore the transferability of solutions found by visual prompt tuning, and find the perturbations are specialized to the model, and anchor prompt.

---

Published as a conference paper at ICLR 2025
Exemplar Images
Task (L.., Generation)
Task (i.e., Detection)
Text Encoder
Text Encoder B
" <dog>> "
" <dog>>
Optimizer
<dog>> "
T(v)
New Word
Transfer Function
More Tasks
Figure I: Large multimodal models can learn specific concepts, such as black Labrador retriever in the figure, after fine-tuning just the prompt. How do soft prompts obtained via prompt tuning methods that encode specific visual concepts (i.c. black dog) differ from typical discrete prompts? We study the solutions found by prompt tuning methods three models, and show their solutions akin adversarial attacks text encoders.
Indeed, the perturbations found by visual prompt tuning are akin to adversarial attacks (Goodfellow al. 2015 Brown et al. 2017), on the embedding space of the text encoder.
This work contributes a large-scale study of soft prompts that encode specific visual concepts across generation, detection, and classification tasks. We provide benchmark for training such prompts on diverse set of visual concepts, and evaluating their transferability across three models. Our work aims to galvanize the interoperability of large multimodal models following Figure [ allow- ing prompts trained for generating black Labradors to be re-used for detection, and other tasks. Transferring prompts can significantly improve the adaptability and cost machine learning sys- tems by eliminating the need re-train prompts when new models released. We highlight the difficulty of transferring prompts for current models, investigate what these prompts learn, and what prevents them from transferring successfully. Code for reproducing work, and benchmarking new transfer methods is available at the following official website: wayward-conceets
RELATED WORKS
Text-To-Image Generation. With the advent of diffusion- architectures, large-scale gener- ative models have developed impressive photo-realism. Approaches like Stable Diffusion (Rom-) bach et al., 2022). DALL-E (Ramesh et al. 2022), and Imagen (Saharia et al., 2022) employ diffusion based approaches (Ho et al.) 2020; Sohl-Dickstein et al. 2015) that start from initial Gaussian noise map, and iteratively denoise the image over several denoising diffusion steps. These approaches incorporate pretrained text-encoders, such as CLIP (Radford et al. 2021 in Stable Dif- fusion (Rombach et al., 2022), to guide generation in the diffusion process. Guidance is typically applied through Classifter-free Guidance (Ho & Salimans 2022), which allows the influence of the text-encoder to be increased, at the expense of generation quality. Diffusion models have remark- able flexibility, and can generate new subjects from handful of examples by learning embeddings for pseudo tokens representing the subject in the prompt GGal et al. 2023 Trabucco et al.) 2023). Fine-tuning both the model and the prompt, as in Dreambooth (Ruiz et al.]] (023). leads to improved generation of subjects, while retaining the controllability of pseudo tokens. Pseudo tokens in diffu- sion models are an application of visual prompt tuning, and our investigation considers them.
Open-Vocabulle Vocabulary Object Detection. Parallel to work in generation, large-scale object detection models have developed comparable strong versatility, and detect new objects from short de- scriptions of their appearance (i.e. detect black dog) (Zou et al. 2023b Liu et al.] [203bb Minderer Models like Grounding DINO (Liu al. 2023b), SEEM (Zou al.] 2023b), and
Open-
et al. 2022
2

---

Published as a conference paper at ICLR 2025
OWLv2 (Minderer et al. sifying bounding boxes. In OWLv2 (Minderer et al.] 2022), representations from a pretrained CLIP (Radford et al. 2021) text encoder are contrasted with region-base' representations from vision transformer backhone. Grounding DINO (Liu et al. 2023b), and SEEM (Zou et al.) 2023b) em- ploy representations from pretrained text encoder (BERT (Devlin al. 2019), and UniCL (Yang) et al. 2022), respectively) to directly guide bounding box proposal. We show open open-vocabulary ob- ject detectors can detect new objects from a handful of examples by optimizing just new prompt embeddings for the object. Our analysis shows key properties of these optimized prompts in open- vocabulary object detection are shared with text-to-image generation.
2022) employ a pretrained text encoder to produce representations for clas-
Zero-Shot Classification. We use recent models (Fang et al. 2023) derived from CLIP (Radford eet al. 2021) for zero-shot classification experiments. We insert new word embeddings optimized for classifying new visual concepts in the prompt of the CLIP text encoder, and contrast text rep- resentations with image representations from the CLIP vision encoder on test images. Prior work
shows CLIP is an effective zero-shot classifier (Radford et al. [202]] Novack et al., 2023). We use checkpoints from OpenAI CLIP (Radford et al.] [2021). OpenCLIP (Cherti et al. [202)) and Data Filtering Networks (Fang et al., 2023), trained on LAION-50 -BB (Schuhmann et al. 20222. Diffusion
models can also be used as zero-shot classifiers (Li al. 2023 Clark& & Jaini 2023). but we focus
on CLIP for better task coverage. Our analysis shows that CLIP has a fractured embedding space, and visual prompt tuning for zero-shot classification finds fractured solutions.
Prompt-Tuning. The word embeddings we optimize for visual concept learning tasks are closely related to prompt-tuning (Lester et al. 2021 Li & Liang] 202T). Prompt tuning aims to find prefix or an entire prompt that causes a pretrained language model to perform a specialized task, such as reading tables (Lester et al. 2021 Li & Liang 2021). These methods treat the prompt as a trainable parameter, and optimize the embeddings of the prompt to minimize a task loss function. Prior work has shown the resulting soft prompts in language modelling tasks are hard to interpret (Khashabi et al., (2022), as their closest discrete prompts are often unrelated to the desired task. Transferring learned prompts is an important task in jail-breaking LLMs (Zou et al. 2023a Robey et al., 2023). and researchers
searching over discrete prompts (Wen et al.] [023 Shin et al.) 2020; Zou et al.)
2023a Robey et al., (023). In pure language modelling tasks, researchers have shown that certain soft prompts can transfer between models with the same architecture and task, but different weights (Passigan et al. 2023: Ju et al. 2023 Wu et al. 2023). We extend this investigation to visual tasks, models with different label modalities (images, bounding boxes, and class labels).
Adversarial Examples. Perturbations found by visual prompt tuning near discrete prompts are akin to adversarial attacks on the embeddings of the text encoder. Adversarial robustness is an tensively studied field in computer vision (Goodfellow et al. 2015), with a variety of attack methods,
including (Goodfellow al. 2015 Brown et al. 2017 Madry al.] 2018 Kurakin et al.) 2017 Car-
lini & Wagner] (017)) and defense methods, including (Madry et al. 2018) Oin et al.] 2019: Tramèr a 2018] et al. 20118). Adversarial attacks in computer vision traditionally focus on mod- ifying the pixels in an image, whereas we modify word embeddings. Adversarial attacks on language include jail-breaking (Zou et al. [202al] Robey et al.) 2023), and typically involve searching over discrete prompts (Zhang et al. 2020 Li et al. 2020), rather than continuous embeddings.
TRANSFER EVALUATION METHODOLOGY
Transferring prompt tuning solutions involves finding map between the embedding spaces of dif- ferent models. We call this mapping the Transfer Function T(v), depicted in Figure The Transfer Function maps word representations for visual concepts from the vector space X Rdz for word embeddings in one model, to the vector space ) = 4d. for word embeddings in another model. Space X' may correspond to Stable Diffusson Rombach et al. (2022) word embeddings for a gen- cration task, and y may be OWL-v2 Minderer M et al. (2022) word embeddings for a detection task. Given these vector spaces, the Transfer Function predicts the representation #(w) in the vector space X' for a word w originally from the vector space y given just the word vector representation (w))
T リー : y - X arg min Ew-p [[((w) T()lw)
(1)
T
3

---

Published as a conference paper at ICLR 2025
Transferring Generation - Detection
Prodict Notse
T(τ)
Encoder A
Encoder
Instances
photo of sorange-car-
<orange-cat>
T gen gen-
Ugen
det
Exemplar Images
Nokky Image
Test Image
Figure 2: Transferring prompt embeddings from generation to detection tasks. We fine-tune the vector embed- dings for tokens (such <orange- for the orange in the figure) minimize noise prediction loss generation. Vector embeddings are transferred from generation to detection using linear map, and used to produce zero-shot instance detections for the target visual concept (in this case, orange cats).
The Transfer Function T(v) minimizes the average prediction error between transferred word em- beddings (w) and real word embeddings z(w) from the vector space X. We average this error over uniform distribution Pa of words that exist in both vector spaces X' and V In our experiments on Stable Diffusion 2.1 Rombach et (2022) and OWL-v2 Minderer al. (2022), the number of
words in Pan (> 40,000) is larger than the number of components d, and dy in each vector space.
3.1 FINDING A LINEAR TRANSFER FUNCTION
Solving the optimization problem given by Equation is hard in general, and to simplify the investi- gation, we restrict focus the class of linear Transfer Functions. This restriction transforms the hard problem Equation iinto a Linear Least Squares estimator, which has closed-form solution. Consider a pair of matrices X E Raxdz and Y E Rexd,, where each pair of rows in X and Y is pair of word vector embeddings ((w) and 7(w) for word w contained in the support of the distribution Pw. The Linear Least Squares estimator we employ for T vーv is given below.
T Ty= arg min Eurep. [[F(w) TT)) ::::==xY>>> (YTY)-'''
One can interpret T v as lining up the directions in the vector spaces X' and y that correspond to the same visual concepts. Word embeddings often have algebraic relationships (Mikolov et al., 2013), and linear Transfer Function preserves these relationships by distributing over addition.
3.2 EVALUATING PROMPTS ON TRANSFERRED TASKS
Using Equation 2 estimate Transfer Functions between all six ordered subsets of the three mod- els, and evaluate prompts optimized for visual concepts on one task (such as generation), and trans- ferred to the same visual concepts another task (such classification). Consider dataset D of images depicting a specific visual concept, such as black Labrador retriever, and task-specific annotations ay such bounding boxes (a,, R or class labels (a., N). We first optimize the prompt embeddings J. E y to minimize task-specific loss function Ly. We then zero-shot transfer embeddings J; task using the linear map Uh T y wーン vー T. and evaluate a task-specific perfor- mance metric Mz. Loss functions and performance metrics for each task are shown in Table []]
E ELa M2 U 1,0z) S.t. T7, = arg min E,, Eayay Cy L,ay)
For Stable Diffusion 2.1 (Rombach et al. 2022). we use the denoising loss function originally pro- posed in al. 2020), where the goal is to predict noise map added image / at particular
4

---

Published as a conference paper at ICLR 2025
Task
Loss Function
Performance Metric
Generation
E E ||e - ca aval a + VI ate,t, 2)
(-- 1[ [[ has the concept
Detection
E B,,,wwDDmi [w (Eabjeat (I, (, (x))
Mean Average Precision
Classification
E BI, Dran
[w (Cimage (1)² Exex (J))
Classifier Accuracy
Loss Functions and Performance Metries. We benchmark transfer of prompt embeddings across generation, detection, and classification tasks. In each row, 7 corresponds to an image, b to an object bounding hox, and w E { to weight multiplied onto the loss function. This weight controls whether the objective maximized or minimized, where otherwise. The functions Z are image and text encoders that return vector representations: Eiaage is the CLIP vision encoder, Tea is the CLIP text encoder, and Eatjent is an OWL-v2 region proposal feature.
Table 1:
-1 when the image and bounding box contain the target concept, and
timestep in the diffusion process We optimize the prompt embeddings Ü so that Stable Diffusion generates images of a particular class (such as black Labrador). This optimization uses training dataset Durain and a separate dataset Diest that contains different images of the same visual concept measure the prob-
(such as black Labrador) is used for evaluation. For evaluating generation, ability that generations contain the target visual concept, measured by OpenAI's pretrained CLIP L-14 model given the prompt "a photo of {sub ject.name))" This procedure employs Textual Inversion (Gal et al. 2023) with the additional prompt embedding transfer step in Figure 2
We standard loss functions and performance metrics adapted from recent literature when training and evaluating visual prompts. Each loss function and performance metric is discussed further in Section 4.3 Now equipped for training, evaluating, and transferring prompts,
pose motivating question: how do solutions from prompt tuning differ typical discrete prompts?
PROMPT TUNING FINDS FRACTURED SOLUTIONS
Near any discrete prompt in the embedding space of Large Multimodal Models, there are perturba- tions that reprogram models to generate, detect, and classify arbitrary subjects. For example, the prompt tuning solution in the top-left of Figure 3 is closest to the discrete prompt "cat", but Stable Diffusion generates vase. We observe this high degree of model reprogrammability consistently across three model classes, four standard datasets, and 40 diverse visual concepts, suggesting a gen- eral phenomenon in Large Multimodal Models. We refer to this phenomenon fracturing of the embedding space, as prompts that encode (i.e. generate) specific visual concepts are scattered across the entire embedding space. Figure shows example solutions -ech row corresponds to a visual concept from a standard dataset, and each column is a discrete anchor prompt. In several cases, an identical image is generated by several perturbations near different anchors, such as generations for the duck concept (second row) from the vase (column two) and candle anchors (column four).
4.1 DATASET PREPARATION
We explore the fracturing phenomenon using four standard datasets, adapted from recent literature in generation, detection, and classification. We adapt the 2014 ImageNet detection dataset (Deng let al. 2009). the DroamBooth dataset (Ruiz et al., (023). COCO (Lin et al.) 2014). and PASCAL VOC (Everingham et al., 2010). For each dataset, we select 10 concepts uniformly random from available classes to use for benchmarking, and select 8 images per concept from the training set (see Appendix D). These cover a diverse range concepts likely to be encountered in real use cases. For ImageNet, each image is annotated with an integer class label, and a set of bounding boxes that contain the target concept. For the DreamBooth dataset, bounding box labels are missing. To obtain bounding box labels, we pretrained OWL-v2 on every image using the name of the subject as the prompt, and manually verified the labels as correct. For COCO and PASCAL VOC, class labels present, assign each image a class label equal the class of the largest bounding box.
5

---

Published as a conference paper at ICLR 2025
csil
yese
dock
candle
unokke
backguck
plusk
beot
clock
sunglaocs
cdleria
Iegtop
deg
Figure 3: Example generations and detections for various concepts (row labels) using solutions found in immediate neighborhood of unrelated words (column labels). We consistently find soft prompts for generating and detecting arbitrary concepts near unrelated discrete prompt anchors across ImageNet (examples in Ap- pendix D}. DreamBooth (first three rows), COCO (last two rows), and PASCAL VOC (Appendix) ! datasets. The same objects are detected, and several cases, near-identical images are generated for different anchors.
4.2 MODEL DETAILS
We select three state-of-the-art models in text-to-image generation, open-set object detection, and zero-shot classification. Each model accepts text-based prompt input, containing the prompt embeddings to be optimized (i.e. "<dog>" for the dog concept). For generation, we select Stable Diffusion 2. latent diffusion model proposed by |Rombach al. (2022), For detection, we select OWL-v2 (Minderer et al. 2022), a two-stage object detection model with a region proposal stage, and classification stage that labels region proposals with classes. For classification,
select Data Filtering Networks (Fang et al. 2023). which use CLIP-based engguage-image contrastive learning (Radford al. 2021)
filtered dataset. These models have different image size requirements. We resize images to 768x768 pixels for Stable Diffusion (Rombach et al. 2022). 960x960 pixels for OWL-v2 [Minderer et al. 2022). and 224x224 for Data Filtering Networks (Fang et al.) 2023).
4.3 EXPERIMENT DETAILS
Training We take all combinations of models, datasets, and concepts, and perform 10 randomized trials varying the discrete prompt anchor to initialize prompt tuning. Anchors are selected as the closest single token the model's tokenizer to the name of the target anchor concept. For example, "sombrero" tokenizes to multiple subwords, experiments account for broad range of initializations. We optimize the prompt embeddings for each concept using the Adam (Kingma & Ba 2015) optimizer with learning rate of 0.0001, and a batch size of 8 (these hyperparameters are shared across all models). We train for 1000 gradient descent steps, and report performance metrics using the final optimized prompt embeddings.
use 'hat' as the anchor. This choice ensures the

---

Published as a conference paper at ICLR 2025
NmtgeNet
DreamBooth
COCO
PASCAL
Generation
0:00
0.50
0:22
0.20
0.22
0.25
6:0
0000
6:75
1.75
Mus Average
0:00
0.00
0.50
0.99
0:2
010000
0.22
0001000
0.22
1110
0.25
600
Clasitier Accurncy
1.00
4:72
0272
0:50
0.50
0.50
0.50
0:25
0025
0.1
0.1
02
02 Distancr From Initislication
Distance From Initialieation
Distance From Initiolication
Distance From Initidiiaation
Trainal For Generation
Trained For Delection
Trained For Chovification
Figure 4: Performance (y-axis) of soft prompts optimized to cause generation, detection, and classification of new visual concepts, for different constraint levels (x- (x-axis). In in-domain performance saturates at constraint level of 8 0.5, which corresponds to solutions where the nearest existing word vector is the anchor Wanchur- < 1.
Constrained solutions perform well in-domain, hut typically don't perform well on transferred tasks Each line in the figure corresponds to the 95% confidence interval of 100 randomized trials for 10 concepts, and 10 anchor words per dataset. Refer to Appendix(D for the concepts and anchor words used for each dataset.
Loss Functions For generation, we employ the standard reparameterized denoising objective, in- troduced by Ho et al. in DDPM (Ho et al. 2020)p For detection, we maximize the cosine similarity between the text and region feature containing the target object, and minimize cosine similarity to all other region features proposed by OWL-v2 (Minderer et al. 2022) in the image. For classification, we maximize cosine similarity between text and images of the target concept, and minimize cosine similarity to images that don't 't contain the target concept. Table shows the exact loss definitions.
Metrics For evaluating the quality of generations, report the rate at which a pretrained OpenAI CLIP L-14 (Radford et al. 2021) classifier predicts that generations are the target class, labeled Generation Accuracy in Table I The label set for Generation Accuracy is the set of all concepts from the corresponding dataset (names listed in Appendix D). For detection, we report the Mean Average Precision of OWL-v2 (Minderer et al., 2022) bounding box predictions on held-out validation set. For classification, we report the accuracy of DFN CLIP (Fang et al. 2023 Radford et al. (02T) given images of all concepts (both positive and negative examples) from a held-out validation set. All metrics are reported as 95% confidence intervals over 10 randomized trials varying the anchor.
4.4 CONTROLLING THE LOCATION OF SOLUTIONS
Large Multimodal Models are highly reprogrammable. We explore this phenomenon by considering a constrained objective for soft prompts in Equation 4 where given an anchor token Wanchor used to initialize V. and a threshold 6 normalized by the distance of the closest discrete token to the anchor, we constrain solutions for to an 12-ball of radius implemented using projected gradient descent. We conduct large-scale experiment, optimizing 4,800 prompt embeddings for 40 visual concepts across four standard datasets, three models, and four constraint thresholds å E (..1, 0.5, 1.0].
T y Wanchor)
Fy = arg min Er,ay (T I, S.t. min or (v(W) y ( (anchnor) 1|
5
(4)
7

---

Published as a conference paper at ICLR 2025
Net
Booth
COCO
PASCAL
Anchor Rde sntevract
0.90
0:2
0.50
035
146
6:00
0.25
0.50
6.75
100
600
0.50
0.75
1,00
200
6222
0.30
075
100
Geseration Accuracy
Generatin Accursey
Generation Accuracy
Generation Accuracy
Mean Average Precision
Menn Average Prediion
Areaarr Precision
MecaaAveragy Precision
100
6:00
0.20
6AB
Classifee Accuracy
Classtfier Accuracy
LlootfferAAcurma
Chantfier Accurayy
Optimal Prompt Anchor
Random Prompt
Figure 5: Performance (x-axis) of soft prompts optimized cause generation, detection, and classification of new visual concepts under a constraint level of 6 0.5, split by the relevance the prompt anchor (y-axis). Optimal prompt anchor corresponds to selecting the target concept name (i.c. "cat") as the anchor, and random prompt anchor corresponds to the mean performance of all other unrelated concept names (see Appendix [ Fach bar corresponds to the 95% confidence interval of the corresponding metric across the 10 target concepts. We observe no impact of the relevance of the prompt anchor to the target concept in task performance.
This experiment controls the location of solutions found by prompt tuning in the embedding space, help us understand the relationship between location, and fidelity of the solution. Using this tool, we will ask: where are the performant solutions to prompt tuning found in the embedding space?
Abundance of Performant Solutions We measure the performance of solutions from prompt tun- ing under varying constraint levels 5 10 0.2, 0.5, 1.0), and discrete prompt anchors. Figure shows a consistent behavior across models, and datasets that performance saturates at a constraint level of 6 0.5, when the nearest neighbor is still the anchor Wanehor- For constraint levels performance does not improve further, despite the larger solution space. Solutions lose a consider- able degree of performance when transferred, which suggests the perturbations are modeel-peecillized and strikingly not emergent embeddings akin to the typical discrete prompts. Instead, performant solutions to prompt tuning are closer to adversarial perturbations than traditional prompts, and appear to be found in the vicinity of most discrete prompts in the embedding space.
1,
Tyy y anabee)))
Resilience to Anchor Points We explore the relationship between the relevance of the discrete prompt anchor, and the target concept to performance in Figure 5 and observe no major impact. We take the solutions found under constraint level of 5 = 0.5, and stratify performance based on whether the discrete prompt anchor related to the target concept. Despite intuition suggesting they would require more optimization to modify the default behavior of the model, solutions anchored to random discrete prompts perform just as well as those constrained to the optimal related anchor. The surprising resilience of prompt tuning solutions to their anchor point suggests the underlying model has such a high degree of reprogrammability that a perturbation can be found that targets any discrete prompt in the embedding space and overrides the model's 's original behavior for that prompt.
4.5 SOLUTIONS TARGET SPECIFIC LAYERS
Performant solutions are numerous in the embedding space, and these solutions are specialized. How can we tell these solutions apart from typical discrete prompts? One characteristic that iden- tifies prompt tuning solutions their effect on the representations predicted by the text encoder. Solutions following those in Figure 4 target the final layers of the text encoder, and steer the pre- dicted representation towards the target concept. Figure sssows generations from Stable Diffusion 2.1 |Rombach et al.) (2022) when truncating the text encoder to just the first N transformer blocks (block Norm + Attention -> Residual >> Norm -> MLP -> Residual). The bottom row shows TSNE visualizations of the pooling token representation at four evenly spaced layers in the text encoder of Stable Diffusion when generating concepts from the ImageNet (Deng et al., 2009) task. The representations initially cluster around strawberry, the anchor concept, and generations from early layers yield strawberries instead of the target concept, sombrero. As we probe deeper layers in the text encoder, representations are gradually steered towards the target concept by the final layer. This can be observed by the evolution of TSNE clusters towards a full separation by color, and the change in generations from the initial strawberry to the final sombrero by layer 24.
8

---

Published as a conference paper at ICLR 2025
Anchor Word: " strawberry "
Prompt: " photo of a <sombrero> "
X
x
X
Mismatched
Evolution of Text Encoder Activations
Fully Separated
Text Encoder Layer
Text Encoder Laver
Text Encoder
Text Encoder Layer 24
Mergea
Figure 6: Prompt tuning solutions target the final layers in text encoders. We show images generated by Stable Diffusion when truncating the text encoder the first N layers, and create TSNE visualizations of the text encoder representations different visual concept. Representations in plots 1-16 cluster around anchors instead of the target concept. When truncating the text encoder to just these layers, the anchor concept (strawberry) is generated instead of gradually steered
the pooling token at evenly spaced layers. Each color represents
the target concept (sombrero). As probe deeper into the text encoder, representations away from the anchor and towards the target concept so that clusters and generations are correct by layer 24. Additional visualizations in Appendix show clusters for the remaining models and datasets.
Understanding The Results The results suggest that soft prompts obtained via prompt tuning methods that encode specific visual concepts are more akin to adversarial perturbations than tradi- tional discrete prompt embeddings. Solutions can be found in the close neighhorhood of any discrete prompt that reprogram the model to generate, detect, classify an arbitrary target concept. Prompt tuning solutions commandeer the final layers of text encoders, and steer the representations towards the target concept. The ease of finding performant solutions independent of their initialization, and anchor point suggests that Large Multimodal Models are highly reprogrammable.
5 SEMANTIC SIMILARITY OF INPUT EMBEDDINGS
To contextualize these results, we conduct an analysis to explore the degree semantic similarity of the input embeddings of tested models. We seek to confirm that the input embeddings share high degree of semantic similarity for these models, despite being training on different tasks, and to illustrate that the non-transferability of prompt-tuning solutions arises due adversarial behavior. To begin, we will show that embeddings for discrete prompts, and randomly sampled embeddings in the neighborhood of discrete prompts are transferable between these models.
5.1 NON-TUNED EMBEDDINGS ARE TRANSFERABLE
To show that transferable embeddings exist, we lookup embeddings for the anchor word of each visual concept, and compare the performance of generation, detection, and classification with the anchor word embedding, to generation, detection, and classification with transferred embeddings. Importantly, when constructing the Transfer Function based on Equation 2 for this experiment, we hold our the set of anchor words as a test set. Shown Figure 7 embeddings for anchor words are transferable in nearly every example, and nearly match the performance of the in-domain prompt. minimal reduction in performance is observed when transferring embeddings to detection, but the resulting performance is significantly higher than transferred prompt-tuning solutions. A secondary experiment exploring the transferability randomly sampled embeddings the neighborhood of these discrete prompt is shown in Appendix H and shows similar result.

---

Published as a conference paper at ICLR 2025
ImageNet
DreamBooth
COCO
PASCAL
Doman
0.1
65
06
03
66
04
03
01
48
10
00
0.
65
Accuracy
Acourncy
Accuracy
Accuracy
Domaia
10
Mean Average Preciion
MeaaAversug Average Precision
Mcan Average Precision
Mein Average Precision
Demain
1/4
04
666
0.1
1.0
Classifier Accuracy
Chestfier Accuracy
Claodiier Accuracy
Clandiler Accuracy
Transfer From Geacratien
Tramser From Detection
Truasfer Classifiationn
Figure 7: Transfer results embeddings of anchor words. In nearly all cases, embeddings existing discrete prompts can transferred from the input embedding space of one model to another, and nearly match the in-domain performance. Base models exhibit high degree of semantic similarity in their input embeddings.
Task A
Task B
CKA (RBF)
Task A
Task B
CKA (Linear)
generation
detection
0.636
generation generation detection
detection
0,5
generation classification
0.759
classification
0.633
detection
classification
0.626
classification
0.506
Table 2: Values for the CKA metric between input embeddings. Using an RBF, and Linear kernel, measure the centered kernel alignment score of the embeddings for shared tokens in the text encoders of large multimodal models. Values for CKA range from 0.5 to 0.75, which indicates a relatively high degree of semantic similarity in the structure of the input embeddings.
5.2 EMBEDDING SPACES ARE SIMILARLY STRUCTURED
The transferability discrete prompts, and randomly sampled prompts their neighborhood, sug- gests that large multimodal models acquire similarly structured input embedding spaces. To confirm this hypothesis, we measure the centered kernel alignment score (Korbblihh et al., 2019) for the embeddings of shared tokens in the text encoders of these models. We report scores in Table 2 Based on ranges presented in Kornblith et al. (2019), the measured values appear relatively high, which perhaps makes the fractured-ness of prompt-tuning solutions more surprising than otherwise. Transferable embeddings for visual concepts do exist, but prompt tuning struggles to find them.
6 DISCUSSION
How do soft prompts obtained via prompt tuning methods that encode specific visual concepts (i.e. black dog) differ from typical discrete prompts? We conduct large-scale study of prompt embed- dings that encode specific visual concepts across generation, detection, and classification tasks, and show that prompt tuning solutions are akin to adversarial attacks on text encoders. Our results sug- gest that I Large Multimodal Models have fractured embedding spaces, where perturbations can be found within an e-ball to any discrete prompt that reprogram the behavior of the underlying model. The ease of finding these perturbations for any discrete prompt with performance comparable to an optimal initialization suggests that Large Multimodal Models are highly reprogrammable in general. One consequence of these findings is that, without careful regularization, is unlikely that solutions found by prompt tuning in Large Multimodal Models will be interpretable to another model.
Our work aims to galvanize the interpretability of solutions found via prompt tuming by exploring their relationship to discrete prompts, and their affect on the underlying mark for evaluating the transferability of soft prompts enable researchers further study and improve the interoperability of prompts following Figure allowing prompts trained for generating black Labradors be re-used for detection, and other tasks. Transferring prompts can significantly improve the adaptability and cost of machine learning systems by eliminating the need to re-train when new models are released. We highlight the difficulty of transferring soft prompts for current models, and our analysis begins explain why transferring soft prompts is so challenging.
We provide a bench-
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Tom B. Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer. Adversarial patch.
CoRR, abs/1712.09665, 2017. URL http:://axiii/ org/ abs/1712 http:p///aaxii.oooraa////22222.0 09665
Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA, USA, May 22-26, 2017, pp. 39 57. IEEE Computer Society, 2017. doi: 10.1 199/S...22.. URL https://doi.
org 10. 1109/52. 0tt.:/5.2017.. 2017.49
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilhareo, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-iman image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition, pp. 2818-2829, 2023.
Kevin Clark and Priyank Jaini. Text-to-image diffusion models are zero shot classifiers. In Al- ice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: www.ceee3cad3a0222655533323 -t Conntet-ooffeeeeeeee html
n
/dlle Abstract
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009, doi: 10.1 10111///./200000 0200944.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre- pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Compulational Linguisties: Human Language Technologies, NAACL-HIT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics, 2019. doi: 10.18653/V 1423. URL htt::://doi..oo TEtp:://0o..02g/10111883/
1423
M. Everingham, L. Van Gool, K. Williams, J. Winn, and Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303-338, June 2010.
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks, 2023.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL htRt:h/pp::///t /otttpsi///ppeneviiee. net/pdf. dndffi 08TcyG
lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceed- ings, 2015. URL http:://axxi..rgg//// 1141. 6572
Jonathan Ho and Tim Salimans. Classifier-free diffussio guidance. CoRR, abs/2207. 2022. doi: 10.48550/AXXI.2227.11555 URL /tttts:///ooo/t 12598
100 20t55/10...855/aaaxxx....
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Mare' Aurelio Ranzato. Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL http://tss./ppooeeee.......eee.. httpa//sss...c/aaeee////////ss/

---

Published as a conference paper at ICLR 2025
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis, 2024.
Tianjie Ju, Yubin Zheng. Hanyi Wang, Haodong Zhao, and Gongshen Liu. Is continuous prompt combination of discrete prompts? towards a novel view for interpreting continuous prompts. In Anna Rogers. Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023. pp. 7804-7819. Association for Computational Linguisties, 2023, doi: 10. 10.18653// 1/202311122331 110.5WWWG22. FINDINGS-ACLL.44 fi Findings-aane acl 494
URL https::///// /doi oogg//org/0000 1 8 8653/v3322 htt/v/122022 /2023.
pettps::/doo..oo
Harini Kannan, Alexey Kurakin, and Ian J. Goodfellow. Adversarial logit pairing. CoRR,
abs/ Nos//803306373, 2018. Utthttp::/ wtwivviiiivv.oo//avvo org/ abs/ 100pp//833303333
Daniel Khashabi, Xinxi Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sean Welleck, Hannaneh Hajishirzi, Tushar Khot, Ashish Sabharwal, Sameer Singh, and Yejin Choi. Prompt waywardness: The curious case of discretized interpretation of continuous prompts. In Marine Carpuat, Marie- Catherine de Marneffe, and Iván Vladimir Meza Ruíz f. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan- guage Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pp. 3631-3643. Association for Computational Linguistics, 2022. doi: 10. 18653/VV/20222 /ttp:///222222211112.. NAACL-MAIN. URL
or.. 10. 18..1.18888 0655/v1/20000 naac ma in 26
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.). 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URLhttp: //arxiv.org///
11122 6980
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and outputs. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaïi, USA, volume 202 of Proceedings of Machine Learning Research, pp. 17283-17300 PMLR, 2023. URL https: //proceedings mll press wwwp/2222222222
html
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E, Hinton. Similarity of neural network representations revisited. CoRR, Abs//1955.0041, 2019. URL http:://axiv..rrt
abs/1905. 00414
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net, 2017. https: //openzeview. net hwtp::/ee/etttooeeeeeeo http://ooommiidtu-----
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for pfrfmeter-efficient prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 3045- 3059. Association for Computational Linguistics, 2021. doi: 10. 10.18653/V110 1/ 1/2011.1/222122212 EMWLLE 243. htthhttp::/ //tp:::://do..oo or.. 10. www11/66611// wttp///2211122 http:/222...mmmmmmmma..m.......a 243
Alexander C. Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your difffusion model is secretly a zero-shot classifier. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 2206-2217. IEEE, 2023. doi: 10. N0.1109/CCVV5007022223.33 2.23.00211
URL /httts:://oo..... 11t0////11.110///11111..........
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. BERT-ATTACK: ad- versarial attack against BERT using BERT. In Bonnie Webber, Trevor Cohn, Yulan He, and
Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing. EMNLP 2020, Online, November 16-20, 2020, pp. 6193 0202. Associa- tion for Computational Linguistics, 2020. doi: 10. /tt0////220000 EMNLPP..MEPPPPP... SWN.OOEANN----MIIOOOOO 5000. URL
https::/dob www.doo..org//...0 011.0o.166333111/0// emr p-ma in 500
12

---

Published as a conference paper at ICLR 2025
Xiang Lisa Li and Perey Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li. and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACI/IJCNLP 2021, (Volume I: Long Pa- pers), Virtual Event, August 1-6, 2021, pp. 4582 4597. Association for Computational Linguis- tics, 2021. doi: 10.18853/V1/2000 111.222222LLLLLN.111120 ACL-LONG.333. URL http:: http:///oo..rr111111888188 v1/2021. acl Long..................... 353
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitmick. Microsoft COCO: common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V,
volume 8693 of Lecture Notes in Computer Science, pp. 740 755. Springer, 2014, doi: 10. 778-3-319-10600 16602-11,448 48. URL https: http:://doooogggg// 10.. 1007/977100 3 319-10602-1
48
Haotian Liu, Chunyuan Li. Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Anmual Conference on Neural Infor- mation Processing Systems 2023, NewrlPS 2023, New Orleans, LA, USA, December 10 16, fi
2023, 2023a.
ni URL otpps ttspppppee
6dcf
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: marrying DINO with grounded pre- training for open-set object detection. CoRR, 0os/2303.05499) 2023b. doi: 10.48550/ARXIV. 2303.05499. URL
/rg//c. 48540 /at.s//axiv...om 2303. 230.3.55399 05499.
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Frozen pretrained transformers as universal computation engines. In Thirty- Sixth AAAI Conference on Artificial Intelligence, AAAI
2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence. IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 March 1, 2022, pp. 7628-7636. AAAI Press, 2022. doi: 10.1609/AAAL
V3617.20729. URL https: htt.:/oo..ogoo////o//oe 10. 1609 /0/0//aaai... sattaai.v3v611 v3617.227777
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, n
Conference Track Proceedings. OpenReview.net, 2018. URL htttt::///ppereeeeeeee
forum? Jz
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. In Yoshua Bengio and Yann LeCun (eds.). 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013. URL http: /attp://arxiv.. org/ abs/ 3781
Matthias Minderer, Alexey A. Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang. Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. Simple open- -pea-vocabulary object de- tection with vision transformers. CoRR, abs/2205 066300 2022. doi: 10.48550/ARXIV.2205. 06230. URL httpt:://////tt http://ooooooooo.../o/// /55t://///....o000005505555/5//0 /..555//aXiiv. 2205. 06230
Zachary Novack, Julian McAuley, Zachary Chase Lipton, and Saurabh Garg. Chils: Zero-shot image classification with hierarchical label sets. In Andreas Krause, Emma Brunskill, Kyunghyun
Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference
Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. //proceeding ngg. ml www.ss.ress...m..... / v202 nww.nkooooo..oo.. html
PMLR, 2023.
URL https:
Pascal Passigan, Kidus Yohannes, and Joshua Pereira. Continuous prompt generation from lin-
ear combination of discrete prompt embeddings. CoRR, abs/2312. 10323, 2023. doi: 10.48550/
ARXIV.2312.10333. URLhttpe h//a:://do// //do .org/100 48550//8855//// 2312. 10323
13

---

Published as a conference paper at ICLR 2025
Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Al- hussein Fawzi, Soham De. Robert Stamforth, and Pushmeet Kohli. Adversarial robust- ness through local linearization. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neu- ral Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada. pp. 13824-13833, 2019. URL https: / http://eeoceeec...e///ee//s neurips. wtt.rips..ccppppee//2200.. hash/ ww..0defd533333m www.eeeee0eeeeeeeeeeeeeeee wtw.or333eeeee. BbsttBaaa-bbtttaccccccaa Abttract.mmmmm
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Ma- rina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Ma- chine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Ma- chine Leurning Research, pp. 8748-8766 PMLR, 2021. URL http:://ppp.t pttt:::/pproceeings.. .ml. press/v139/ http://adfffdddaa..aaaaa html
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents. CoRR, abs/2204.06125, 2022. doi: 10.48550/
ARXIV.2204.06122 JRLhttps: /ttps::/ddooooooo /tt///i.orggg..o/ 48550/ .te.iii..22....0 211.222224..22222 06125
Alexander Robey, Eric Wong, Hamed Hassani, and George J. Pappas. Smoothllm: Defending large
language models against jailbreaking attacks. CoRR, abs/2310.03684, 2023. doi: 10.48550/
ARXIV:2310.06844 BtLhhttss:/ http::::/doo..rg/10/// 1tt.://ooo//...85//////11111 w11.////............00..... 2310.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In JEEF/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR 2022, New Orleans. LA, USA, June 18-24, 2022, pp. 10674-10685. IEEE, 2022. doi: 111.0/CVV5228..222200000 org/10. 1tt////..1100////522562663 2022. 01042
URL pattps://det
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yacl Pritch, Michael Rubinstein, and Kfir Aber- man. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 22500-22510. IEEE, 2023, doi: 10. N0.119//VVP52222.2002 02155. URL httph: doi org/ /10 111./11/P00/222200 2023 02155
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Raphael Gontijo-Lpess, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion mod- els with deep language understanding. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: //tprr/eeeeeereeeeweeeeee net forum -d=
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL / wpppr//eeeeeeeeeeeee/peee hwt.:/eeweeeouum.ecm M377 4VMMMMYY
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. Au- toprompt: Eliciting knowledge from language models with automatically generated prompts. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020. pp. 4222 4235. Association for Computational Linguistics, 2020. doi: 346. URL htttts::p/.....ooo http////..../....../////////////
10.18653/V 10066/3///22200EEMPPPMMMMM emm 1p-mappmaa-- i.. 346
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsu- pervised learning using nonequilibrium thermodynamics. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML
14

---

Published as a conference paper at ICLR 2025
2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceed- ings, pp. 2256 2265. JMLR.org, 2015. URL http::/pprocedlnngs www.iinnn.........mmme..mee mir. www..mmr..p.... milr.peess/337
html
Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective data augmen- tation with diffusion models. CoRR, abs/2302.007444, 2023. doi: 10.48550/ARXXV.222200777
URL https:://o...ogg//// 411.00..5555ra.... arxiv. 2302. 07944
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian J. Goodfellow, Dan Boneh, and Patrick D. McDaniel. Ensemble adversarial training: Attacks and defenses. In 6th International Confer- ence on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://cpenrevier net//
Eorum?id=rk2vBe- R2
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. CoRR, abs/2302.036600 2023. doi: 10. 10.48500ARXX..2300.066666
(RLtttss:///cc org/
10. 1...8555/arXiv/ 2302. 03668
Zijun Wu, Yongkang Wu, and Lili Mou. Zero-shot continuous prompt transfer: Generalizing task semantics across language models. CoRR, abs/2310.01691, 2023. doi: (0.48550/ARXIV.3310 01691. URL /tt.pp://dl..oo//1000555335 wttp://00/...1.....001111
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Uni- fied contrastive learning in image-text-label space. IEEE/CVF Conference on Computer Vision and Putlern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 19141- 19151. IEEE, 2022. doi: LLL110/CCVPR2268.2222...2222 2022. 01857
URL https:://// http:://ooo...gggg
Wei Emma Zhang, Quan Z. Sheng, Ahoud Alhazmi, and Chenliang Li. Adversarial attacks on deep-learning models in natural language processing: A survey, ACM Trans. Intell. Syst. Tech- nol., 11:3442::122:41, 2020. doi: 10.1145/3374211 URL htt.tt::::ci.org/oooott
3374217
Andy Zou, Zifan Wang. J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. CoRR, habs/2307.15044 2023a. doi: 10.48550/ARXIV23077. 15043. htth/ttss::///// //ttts::///o....ot orgp:///////o..///////// 48550/ 2307.15043
Xueyan Zou, Jianwei Yang. Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jian- feng Gao, and Yong Jae Lee. Segment everything everywhere all at once. In Alice Oh, Tris- tan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Ad- vances in Neural Information Processing Systems 36: Annual Conference on Neural Infor- mation Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023b, URL http:////eepss.......cc///aaaaaa Bef6117e44
nww.. httpaapper..illleeppeeeeeeeeeeee
15

---

Published as a conference paper at ICLR 2025
A LIMITATIONS & SAFEGUARDS
We employ pretrained diffusion models, object detectors, and classifiers in this work, and these models are known to have biases, obtained from their training data. Diffusion models in in-particular can generate harmful or dangerous content, including graphic imagery of violence, and pornography. We employ the Stable Diffusion safety checker to flag generations after transferring soft prompis for unsafe content as a mitigation strategy for this potential limitation. Transferring soft prompts currently does not perform very well outside of certain common concepts, and one limitation of this paper is its scope: we do not propose new methodology for transferring soft prompts with high fidelity. Rather, we benchmark popular methods for soft prompt-tuning show that most prompts are not transferable. Our experiments suggest that non non- con-transferable prompts have certain properties that can be used to identify them, but turning this identification strategy into a mitigation method is outside the scope of this paper, and left for future research.
three recent models, and
B ETHICAL CONSIDERATIONS
Diffusion models currently require pristine data showing a subject in clear view in order generate new photos of that subject. Transferring soft prompts from an object detector has the potential to allow for training on less pristine data that shows the subject amidst many distracting objects. One potentially harmful consequence of transfer between object detection models and generative models related to privacy. Individuals that don't upload photos of themselves online are currently protected from their likeness being generated by diffusion models. However, transfer from object detectors to generative models would allow for their likeness to be generated, even when photos only show them in crowded spaces. Likewise, transferring prompts from generation to detection allows for the rapid creation of detectors for specific individuals. This technology could be used by malicious actors track the activity of specific individuals, invading their privacy.
C BROADER IMPACTS
Transferring prompts for specialized tasks significantly improves the adaptability and cost of ma- chine learning systems by removing the need to re-train when new models are released. The cadence of multimodal machine learning is such that new models are released every month, and the state-of- the-art constant flux. Currently, soft prompts trained for older models are discarded when newer models are released, or when the task changes (i.e. classification becomes detection). Enabling the re-use of soft prompts would allow users to download prompts trained by someone else, like plugins, even when the original use-case for that soft prompt was for different task (such generation).
One negative broader impact that results from improved transfcrability is that soft prompts encoding negative and harmful behaviours become easier to and maintain. Currently, harmful prompts be- come obsolete quickly as newer models are released, but once they can be transferred, they become permanent. Mitigation strategies for this risk could involve moderating online databases containing soft prompts to remove ones that perpetuate harmful behaviors, and filtering the outputs of models using the soft prompts to directly remove the harmful content (in the same vein safety checker).
D SELECTED CONCEPTS & ANCHOR WORDS
In this section, we discuss the concepts that were selected from ImageNet Deng et al. (2009). COCO Lin et al. (2014), PASCAL Evveinnhhmn et al. (2010), and the DreamBooth dataset Ruiz et al. (2023). These concepts were selected uniformly at random without replacement from the available classes in each dataset. Ten classes were sampled per dataset in order to reduce the computational complexity of the experiments in the paper (results take 3 days to produce on just 40 visual concepts). These classes cover a diverse set of visual concepts.
On the ImageNet dataset Deng al. (2009), we select [' ('strawberry', 'harp', sturgeon', 'gorilla', "throne' pellican' 'honeycomb', 'barrel', sombrero', 'scuba diver') as target concepts.
16

---

In this section, we enumerate the hyperarameters used in the experiments in the paper. We choose hyperparameters agnostic the model and task, that results in the experiments general, and Published as a conference paper at ICLR 2025
Hyperparameter Name
Hyperparameter Value
Generation Model Name
Stable Diffussin 2.1
Rombach et al. (002
Generation Model HuggingFace ID
stabilityai/si httpa/////allle.e att.sataabllllftteeeeee fuus! 2-
Generation Image Size
768 x 768
Detection Model Name
OWL-v2 Minderer et al. 2022
Detection Model HuggingFace ID
google/owlvz base b eesembeeeeemle.
Detection Image Size
960 x 960
Classification Model Name
Data Filtering Networks Fang et al. (2023
Classification Model HuggingFace ID
apple/DFN2B CLI.- ViT
Classification Image Size
224
224
Examples Per Concept
Embedding Vectors Per Concept
4
Denoising Steps
50
Batch Size
Learning Rate
le-04
Gradient Descent Steps
1000
Optimizer
Adam
Adam Betal
0.9
Adam Beta2
0,999
Adam Epsilon
1c-08
Weight Precision
float16
Table 3: Hyperparameters used in the experiments of the paper. These parameters held constant across all datasets and models. These choices are adapted from relevant prior work.
On the DreamBooth Dataset Ruiz et al. (2023), select c'ca''', 'vase', ' 'duckttoy' ccanll'' colorful.sneaker
bbcckpack.dog', "grey.sloth.plushie fancy-boot', 'clock', 'pink.sunglasses' ] as target concepts.
On the COCO dataset Lin et al. (2014), we select [' ('laptop', 'scissors', 'dnuut' bear', 'cup', 'do', 'bottle', 'umbrella', 'cat', 'remote') ] as target concepts.
On the PASCAL VOC dataset Everingham al. 20010 we select l'airplane', 'bicycle', 'bird', 'boat', 'person', 'train', 'aar' 'aat' horse', cow') target concepts.
In addition to selecting concepts, we select anchor words that tokenize to single token across all of the tested models. These are derived from the above target concepts.
On the ImageNet dataset Deng et al. (2009), we select ['strawberry', 'harp', sturgeon', 'gorilla', 'throne', 'pelican', 'honeycomb', 'barrel', hat'' sccbaa as anchor words.
On the DreamBooth Dataset Ruiz al. (2033, we select ('cat', 'vase', 'duck', "candl'', sneaker', 'backpack's 'plush', 'boot', 'clock', sunglasses') as anchor words.
On the COCO dataset Lin et al. (014), we select ['laptop', 'scissors', 'donut', 'eear', 'cup', 'dog', 'bottle', 'umbrella', 'cat', 'remote') as an- chor words.
the PASCAL VOC dataset Everingham et al. (2010) we select l'airplane', 'bicycle', "bird', "boat', 'person', 'train' ' 'aar' 'aat' horse', 'cow') as anchor words.
E HYPERPARAMETERS
17

---

Published as a conference paper at ICLR 2025
ImageNet
DreamBooth
COCO
PANCAL
Domaia
0.2
10
Generation Arcuracy
Generation Accuracy
Generafan Accuracy
Generatien Accuraty
Domaio
0.
0.2
Meon Average Prechlon
Moan Average Precision
Mean Average Prectston
Menn Average Predision
Domaia
Classifer Accurncy
Clasifier
Classiferr
Chassificr Accuracy
Trainal For Generation
Trained For Detretien
Trained For Chociiccation
Figure 9: Visual prompt embeddings trained for one task (i.e. generation) perform well on that task, but may not perform well when transferred another task (i.e. generation detection). In certain directions, such as classification - generation, transfer works better than others. understand when transfer fails, perform extensive ablations across four standard datascts, and three models generation, detection, and classification.
not specific to the model. Table 3 we note the HuggingFace model ID used, model configuration details, and hyperparameters from training, and evaluation.
F UNCONSTRAINED SOLUTIONS
DreamBooth
PASCAL
We consider a variant of the experiment in Figure where optimize prompt embedding using uncon- strained objective, and evaluate their task performance. We conduct large-scale experiment, training 1200 new prompts for the same 40 visual concepts used in the main paper. We use standard gradient descent, instead of projected gradient descent to calculate gradients, and backpropagate gradients directly into the prompt embed- dings. After training prompt embeddings to optimal per- formance on the training task, we evaluate on different test task using metrics discussed in Section 4.3
Targel Images
Trained For Generation
X
Trained For Classification
X
X
X
F.1 EXPLORING TRANSFERABILITY
Results of the experiment in Figure 9 show that prompts optimized for visual tasks can perform well in-domain, but typically not re-usable. In most transfer scenar- ios, prompts optimized for one task don't solve a different task than they were trained on with comparable fidelity to in-domain training. Prompts optimized for classification transfer best, achieving up 84% of the performance of in-domain training for generation (PASCAL), and up to 28% of the in-domain detection performance (ImageNet). Prompts optimized for detection are least transferable, achieving up to 26% of in-domain classification perfor- mance (COCO), and up to 30% of in-domain generation performance (PASCAL). Prompts optimized for genera- tion are in the middle in terms of their transferability, at- taining up to 59% of the in-domain classification perfor- mance (COCO), and up to 28% of the in-domain detec- tion performance (ImageNet). Generation shows a significant difference in performance between prompts transferred from classification vs. detection, what's happening here?
Trained For Detection
Figure Generations (rows 2-4) from Stable Diffusion for target concepts (top row) from the DreamBooth and PASCAL datascts. The second row trains prompt embeddings for generation. The third row transfers prompt embeddings from classifi- cation to generation. The final row trans- fers from detection. prompts trained for gen- cration capture finc-grain details. prompts trained for classification work for common concepts on PASCAL, but fail at fine-grain concepts DreamBooth. prompts trained for detection generally don't 't transfer.
18

---

Published as a conference paper at ICLR 2025
ImageNet
DreamBooth
COCO
PANCAL
10
Generation Accuracy
Generation Accuracy
Generafam Acturacy
Generatien Accuraty
Domaio
00
0.2
02
Meon Average Prechlon
Moan Average Precision
Mean Average Prectston
Menn Average Predision
Classifie Accurncy
Clasifier
Classifier
Classificr Accurscy
Transfer From Generation
Tramfer Prom Detection
Transfer From Claseiication
Figure 10: Transfer results for randomly sampled embeddings near anchor words. We sample an independent random perturbation for each trial from Normal distribution with mean 0, and standard deviation 0.01. and the embeddings for anchor words Figure These randomly perturbed embeddings show a robustness when transferred, and nearly match the performance of the in-domain discrete prompt.
Understanding The Results Using generation as case study, we show images generated by Stable Diffusion 2.1 in Figure Busing embeddings trained for generation (second row), transferred from classification (third row), and from detection (fourth row). We select two fine-grain concepts from the DreamBooth dataset, and two common concepts from PASCAL VOC. prompt embeddings trained for generation succeed at learning both fine-grain details for subjects in the DreamBooth dataset, and common classes in PASCAL. For prompts trained for classification, however, fine- tne-grain details are missed, but common classes are learned. Results trained for detection miss fine-gra tne-grain details, and common classes when transferred to generation, explaining patterns in Figure
G VERIFYING THE GENERATION ACCURACY METRIC
In the table below, we report the baseline classification accuracies of ther CLIP model used in our Generation Accuracy metric. On each domain, the classifier attains an accuracy of more than 88%, suggesting that it can reliably discern each visual concept in each dataset.
ImageNet
DreamBooth
COCO PASCAL VOC
98.8%
97.5%
88.8%
95.0%
Table 4: Baseline 10-way classification accuracy of the CLIP model selected for evaluating genera- tions from Stable Diffusion 2.1, tested on each domain.
H TRANSFER FOR RANDOMLY-PERTURBE EMBEDDINGS
We continue the analysis from Section 5 and show that randomly sampled prompts in the neighbor- hood of existing discrete prompts maintain performance when transferred. We employ a Gaussian distribution with mean O, and standard deviation 0.01 to sample perturbations, and add these pertur- bations to the embeddings for anchor words used in Figure 7 Results are shown in Figure 10 and illustrate that randomly sampled prompts have comparable transferability to discrete prompts. This suggests the non-transferability of prompt-tuning solutions is due primarily to adversarial behavior.
I MORE COMPLEX TRANSFER FUNCTIONS
In the main text of the paper, explored a linear transfer function, and showed that under such a map, soft prompts optimized for one multimodal model were non-transferable. We provide ad- ditional results with an ML. transfer function in Figure 11 and find consistent behavior with the
19

---

Published as a conference paper at ICLR 2025
ImaggeNet
DreamBooth
COCO
PASCAT.
Grocration Accuurt
0.90
0.24
0.3
0.00
080
190
0.0
1.00
Classterr Averagg
0.90
0:35
0.35
035
0:35
0.90
1.9
0.00
0.50
150
0.25
035
135
0:5
0.90
180
Distence From Initialization
Distonce From Inttialization
Distonce From Inttialization
Distance From Indtialbrarion
Trained For Generation
Trained For Detection
Tratoed For Classficattio
Performance of word vectors (y-axis) optimized for one task, and transferred another model using a two-layer MLP transfer function, for various constraint levels 5 (x-axis). Under an MLP transfer function, transferability of word vectors did not improve.
Figure 11:
linear transfer function. We train a two-layer MLP with ReLU activation functions, and 4096 hidden units to optimize the embedding transfer loss function in Equation [ Transfer performance of soft prompts is marginally worse with this MLP transfer function than with a linear transfer function.
J MORE EXAMPLES
In this section, we show more examples of generations from Stable Diffusion for perturbations to various unrelated anchor words in the embedding space. We show results for all combinations of 10 target concepts (row labels) and 10 anchor words (column labels) on ImageNet Deng et al. (2009). COCO Lin al. (2014), PASCAL (Everinhnnn
al. (2TOO), and the DreamBooth dataset|Ruil Ruiz al. 2003). In several cases, nearly identical images are generated by Stable Diffusion for perturbations near to different unrelated anchor words.
K ADDITIONAL VISUALIZATIONS
We provide additional TSNE visualizations of the text encoder activations for different models and datasets in this section. Patterns in Section 4.5 hold across all tested models and datasets. Pertur- bative soft prompts like those found in Section 4.4 target the final layers text encoders, and early activations in text encoders disagree with later activations. Generating images when truncating the text encoder to the first N layers leads to generations of the anchor, instead of the target concept we optimizing for (see Figure When perturbative solutions transferred, this transition stops.
Fine-tuning that targets the final layers of text encoders does not transfer, and Figure 22 shows that activations stop clustering by concept (color) when soft prompts are transferred.
20

---

Published as a conference paper at ICLR 2025
Loro
Dear
Figure 12: Visualizations of detections from OWL-v2 Minderer et al. (2022) using embeddings
optimized for detecting visual concepts on COCO al. (2014). Performant solutions for detect- ing arbitrary target concepts (row labels) are found with constraint threshold å 0.5 of unrelated anchor words (column labels).
21

---

Published as a conference paper at ICLR 2025
Airllan
Den
Bleck
TH
Figure 13: Visualizations of detections from OWL-v2 Minderer et al. (022) using embeddings optimized for detecting visual concepts on PASCAL Everingham al. (2010). Performant solutions for detecting arbitrary target concepts (row labels) are found with constraint threshold 0.5 of unrelated anchor words (column labels).
22

---

Published as a conference paper at ICLR 2025
Sturkur
Bara
Page
Grdla
Prlcs
Banss
Soda
Pilon
Figure 14: Visualizations of generations from Stable Diffusion 2.1 Rombach al. (2022) using embeddings optimized for generating visual concepts on ImageNet Deng et al. (2009). Performant solutions for generating arbitrary target concepts (row labels) found with constraint threshold 0.5 unrelated anchor words (column labels). In several cases, different solutions far apart generate the same image.
23

---

Published as a conference paper at ICLR 2025
Cd
Yast
Tack
Caudss
Soudiot
Backgack
Fau
Bert
Clock
Sue gagaa
24
Chu
Figure 15: Visualizations of generations from Stable Diffusion 2.1 Rombach al. (2022) using
embeddings optimized for generating visual concepts on DreamBooth Ruiz et al. (2023). Performant solutions for generating arbitrary target concepts (row labels) found with constraint threshold 5 0.5 unrelated anchor words (column labels). In several cases, different solutions far apart generate the same image.
24

---

Layer Published as a conference paper at ICLR 2025
Layer
Layer
Layer
Layer
Figure 16: Visualizations of text encoder activations for OWL-v2 Minderer et al. 2022 on COCO Lin et al. (2014) at four evenly spaced layers when optimizing soft prompts for detecting visual concepts (colored points), constrained to the neighborhood of various anchor tokens (clusters in plots 1-8).
Layer
Layer
Layer
Figure 17: Visualizations of text encoder activations for OWL-v2 Minderer et al. 2022) on PASCAL Everingham et al. (2010) at four evenly spaced layers when optimizing soft prompts for detecting visual concepts (colored points), constrained to the neighborhood of various anchor tokens (clusters in plots 1-8).
Encoder Layer
Encuder
Encoder Layer
Encoder Layer
Figure 18: Visualizations of text encoder activations for DFN CLIP et al. 20233 Radford et al.) (202T) on ImageNet Deng et al. (2009) at four evenly spaced layers when optimizing soft prompts for classifying visual concepts (colored points), constrained to the neighborhood of various anchor tokens (clusters plots 1-8).
Layer
Layer
Layer
Layer
Figure 19: Visualizations of text encoder activations for DFN CLIP et al. 2023 Radford et al.
(202T) on DreamBooth Ruiz et al. (2023) at four evenly spaced layers when optimizing soft prompts for classifying visual concepts (colored points), constrained to the neighborhood of various anchor tokens (clusters plots 1-8).
25

---

Published as a conference paper at ICLR 2025
Text Encoder Layer
Text Encoder Layer 8
Test Enceder Layer 16
Text Encoder Layer
bukkysc
magione
Figure 20: Visualizations of text encoder activations for Stable Diffusion 2.1 Rombach et al. (2022) on DreamBooth Ruiz al. (2023) at four evenly spaced layers when optimizing soft prompts for generating visual concepts (colored points), constrained to the neighborhood various anchor to- kens (clusters in plots 1-16).
Text Encoder Layer
Text Eneoder Layer 8
Text Eneader Layer 16
Text Encoder Layer 24
Figure 21: Visualizations of text encoder activations for Stable Diffusion 2.1 Rombach et al. (2022) on ImageNet Deng et al. (2009) four evenly spaced layers when optimizing soft prompts for gen- erating visual concepts (colored points), constrained to the neighborhood of various anchor tokens (clusters in plots 1-16).
Text Encoder Layer
Text Encoder Layer
Text Enceder
Text Encoder Layer
Figure 22: Visualizations of text encoder activations for Stable Diffusion 2.1 Rombach et (0022) on ImageNet Deng et al.] (2009) at four evenly spaced layers when optimizing soft prompts for classifying visual concepts (colored points) and transferring to generation, constrained the neigh- borhood of various anchor tokens (clusters in plots 1-24). The evolution of clusters towards clean separation for in-domain evaluation stops when soft prompts are transferred. Fine-tuning that targets the original model is lost.
26


Reviews:
[{"review_id": "Ny5Ja4Lrxw", "reviewer": "Reviewer_nh4x", "summary": "This paper examines how fine-tuned prompt embeddings for visual concepts affect text-to-image generation, object detection, and classification. It reveals that these embeddings act as model-specific adversarial perturbations, altering behavior without needing extensive retraining.", "questions": "- The analysis of the transfer function employs a straightforward linear transformation. \n1. Given that X and Y are finite observations, the effectiveness of the linear transformation is influenced by the sample size (n), quality, and representativeness of the samples. Under these circumstances, it is critical to evaluate whether the derived transfer function T is representative and adequately supports the subsequent conclusions. \n2. Considering the potential noise in the observations, it would be worthwhile to explore the use of penalized least squares. Additionally, for linear transformations between two spaces, incorporating sparse regularization could significantly impact the subsequent analysis and conclusions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper provides a novel perspective on prompt tuning across multiple models and tasks.\n- The paper is logically structured, clearly presenting methodologies and findings.", "weaknesses": "- The application of the transfer function and its relationship with Section 4 should be clearly articulated, especially regarding its role in the findings and conclusions of the study.\n- The potential impact of the differences in the datasets used for the experiments on the results and conclusions should be discussed in detail. Currently, the datasets appear to exhibit considerable similarity. The diversity and representativeness of these datasets remain limited. It raises the question of whether the findings from a model trained on large-scale data are genuinely necessary for the conclusions drawn in this study. The application of the findings in domains with greater diversity could yield more valuable insights. A comparative analysis that includes varied domains, such as remote sensing, would enhance our understanding of the generalizability of the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aD7xMz00Lz", "reviewer": "Reviewer_52QW", "summary": "The paper investigates prompt embeddings in large multimodal models, such as Stable Diffusion, to understand how these embeddings differ from traditional discrete prompts for generating and classifying new visual concepts. Through a large-scale analysis across text-to-image generation, object detection, and zero-shot classification, the authors discover that prompts optimized for new concepts function similarly to adversarial attacks on the text encoder. Testing with 4,800 embeddings, they find that these adversarial perturbations specifically target the final layers of text encoders, influencing models to respond to specific subjects, but these effects are model-specific and dependent on initialization.", "questions": "See above weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper presents an interesting perspective for integrating specific concepts into a sequence of multimodal models.\n- The paper proposed straightforward methods for transferring the soft prompts in a source domain into various target tasks, with thorough analysis of its effect.", "weaknesses": "- The paper presumes the linearity between the text embedding space between domains. While models sharing the similar text encoders might be suitable to presume the linear relationship, the models using text encoders with totally different text embedding space might rather collapse when representing the soft prompt of the target domain with linearity.\n- Generalizability of transform function: the paper used 40 visual concepts for transferring experiments, which seem to be limited. Scaling up the visual concepts would be required to see if the transform function can generalize to any of visual concepts.\n- the motivation behind transferring the soft prompt over various tasks: the authors suggested that transferring the soft prompt into other tasks eliminates the need for retraining prompts for each task. Performance comparison between transferring prompt vs prompt-tuning each task would be required to see if the performance gap is negligible while lowering the training overhead.\n- The paper needs re-ordering, where the main goal of why transferring soft prompts is needed stated in the last with a few sentences, which might give not seamlessly connected\n- The paper needs reorganization, as the empirical questions and observations take a large portion of the introduction and abstract in the front, while the main goal—explaining why transferring soft prompts is necessary—is briefly mentioned at the end with a few sentences, which might make the flow disjointed. Also, the details (e.g., derivation of (2)) are often absent, which would rather be better if kindly provided in supplementary materials.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DnL2qjyEYW", "reviewer": "Reviewer_2bvc", "summary": "This work studies the following question: How are prompt embedding for visual concepts found by prompt tuning methods different from typical discrete prompts? It evaluates through transfer learning tasks such as zero-shot detection.", "questions": "See above", "limitations": "", "rating": 5, "confidence": 1, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "The paper is well-written (while I have a hard time to understand the motivation, see below) but the overall presentation is good. Authors did a large-scale analysis on the defined problem as well.", "weaknesses": "I actually have a hard time to understand the motivation of this work, and as a result, my judgement may be incorrect.\n\nSpecifically, it's not clear to me what's the motivation of finetuning prompts like <black_dog> or <orange_cat>? I have seen work doing similar things for personalized generation like DreamBooth but what's actually the motivation for prompts like <black_dog> or <orange_cat>?\n\nFollowing previous point, I don't get the motivation to understand the difference between <black_dog> and \"black dog\" in the embedding space. For what purposes should we care about this? I think the analysis makes sense but it's not clear to me why should we care about this problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9kXgjdIdEJ", "reviewer": "Reviewer_2W9n", "summary": "In this paper, the authors are motivated to learn prompts or tune prompts across different vision tasks. They mostly try to learn prompts for generative tasks and then transfer these learned prompts to discriminative tasks such as classification and detection.", "questions": "1. What if you were to reverse the setup, learn prompts for discriminative tasks and transfer to generative tasks would the results hold?\n2. Did you finetune any layers of the models? To the best of my knowledge, they seemed to have been left frozen.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The idea has practical significance and learning soft prompts that can transfer for various tasks is valuable.\n2. The proposed linear layer learned is simple to understand and the implementation details are properly explained.", "weaknesses": "1. I question the value of this setup if both the generative and discriminative VLMs use the same text encoder. The authors chose Stable Diffusion 2, which uses OpenCLIP L14 for generation, and for classification, they use CLIP L14. However, if OpenCLIP L14 were also used for discriminative tasks, the problem setup might not hold.\n2. Additionally, the paper lacks a discussion on the limitations of their problem setup and proposed methods, which would be helpful for understanding the approach's constraints.\n3. The vocabulary in some sections feels unnecessarily complex and could be simplified for better clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AqZ3ytXAaW", "reviewer": "Reviewer_1THk", "summary": "The paper explores the transferability of prompt embeddings learned through prompt tuning across models trained on different tasks. It concludes that these prompt embeddings are not transferable. The study finds that prompt embeddings function similarly to adversarial perturbations, with multiple effective prompt solutions possible within close proximity to text embeddings of unrelated concepts.", "questions": "1. How semantically similar (measured using CKA or CKNNA) are the input embedding representations? Are the text encoders identical across the three models? If not, do they exhibit high semantic similarity for common vocabulary but not for learned concepts? \n\n2. Why was only MSE loss considered for learning transformations across modalities? Is the failure due to difficulty of transferring between tasks, or could it be attributed to the limitations of using a linear transform or the MSE loss?\n\n3. What happens when the prompt is learnt for one generative model and transferred to another generative model?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Well-written and easy to follow. \n2. Extensive analysis clearly establishes the non-transferability of learned prompt embeddings across models. \n3. Perturbation analysis shows that learned embeddings constrained to random prompt anchors can perform equally well as those near related prompt anchors.", "weaknesses": "1. Identifies that prompt-tuned input embeddings resemble adversarial examples and lack transferability but does not propose mitigation strategies. \n2. Recent works [2] [3] focuses on final-layer convergence for multimodal representations; it’s unclear if this semantic alignment exists in input embeddings. Applying metrics like CKA [1] or CKNNA [2] could reveal input embedding similarities, potentially isolating adversarial behavior as the primary non-transferability factor. \n3. Lacks a control setup in the linear transformation + MSE loss analysis where the transfer works; tuning on one generative model and testing on another could serve as a useful comparison. Is the non transferability because of this Linear+MSE setup?\n4. Limited transformation methods and losses explored—considers only linear transformation and MSE loss. Trying Nonlinear transformations and/or CLIP loss could offer further insights. \n5. Missing citations for recent work on vision and language representation convergence, such as [3] \n\n[1] Kornblith, Simon, et al. \"Similarity of neural network representations revisited.\" International conference on machine learning. PMLR, 2019.\n[2] Huh, Minyoung, et al. \"The platonic representation hypothesis.\" arXiv preprint arXiv:2405.07987 (2024).\n[3] Maniparambil, Mayug, et al. \"Do Vision and Language Encoders Represent the World Similarly?.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\nI am willing to raise my score if more insights are provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)