Wayward Concepts In Multimodal Models

Authors: Brandon Trabucco, Max A Gurinas, Kyle Doherty, Russ Salakhutdinov
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Deep Learning

Abstract:
Large multimodal models such as Stable Diffusion can generate, detect, and classify new visual concepts after optimizing just the prompt. How are prompt embeddings for visual concepts found by prompt tuning methods different from typical discrete prompts? We conduct a large-scale analysis on three state-of-the-art models in text-to-image generation, open-set object detection, and zero-shot classification, and find that prompts optimized to represent new visual concepts are akin to an adversarial attack on the text encoder. Across 4,800 new embeddings trained for 40 diverse visual concepts on four standard datasets, we find perturbations within an $\epsilon$-ball to any prompt that reprogram models to generate, detect, and classify arbitrary subjects. These perturbations target the final-layers in text encoders, and steer pooling tokens towards the subject. We explore the transferability of these prompts, and find that perturbations reprogramming multimodal models are initialization-specific, and model-specific. Code for reproducing our work is available at the following site: https://wayward-concepts.github.io.

Decision: accept (poster)
Reviewer scores: 6, 6, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730045577936

thread_type:
official_review

<thread_object>
creator: reviewer_1thk

<content>
summary:
The paper explores the transferability of prompt embeddings learned through prompt tuning across models trained on different tasks. It concludes that these prompt embeddings are not transferable. The study finds that prompt embeddings function similarly to adversarial perturbations, with multiple effective prompt solutions possible within close proximity to text embeddings of unrelated concepts.

strengths:
1. Well-written and easy to follow. 
2. Extensive analysis clearly establishes the non-transferability of learned prompt embeddings across models. 
3. Perturbation analysis shows that learned embeddings constrained to random prompt anchors can perform equally well as those near related prompt anchors.

weaknesses:
1. Identifies that prompt-tuned input embeddings resemble adversarial examples and lack transferability but does not propose mitigation strategies. 
2. Recent works [2] [3] focuses on final-layer convergence for multimodal representations; it’s unclear if this semantic alignment exists in input embeddings. Applying metrics like CKA [1] or CKNNA [2] could reveal input embedding similarities, potentially isolating adversarial behavior as the primary non-transferability factor. 
3. Lacks a control setup in the linear transformation + MSE loss analysis where the transfer works; tuning on one generative model and testing on another could serve as a useful comparison. Is the non transferability because of this Linear+MSE setup?
4. Limited transformation methods and losses explored—considers only linear transformation and MSE loss. Trying Nonlinear transformations and/or CLIP loss could offer further insights. 
5. Missing citations for recent work on vision and language representation convergence, such as [3] 

[1] Kornblith, Simon, et al. "Similarity of neural network representations revisited." International conference on machine learning. PMLR, 2019.
[2] Huh, Minyoung, et al. "The platonic representation hypothesis." arXiv preprint arXiv:2405.07987 (2024).
[3] Maniparambil, Mayug, et al. "Do Vision and Language Encoders Represent the World Similarly?." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

I am willing to raise my score if more insights are provided.

questions:
1. How semantically similar (measured using CKA or CKNNA) are the input embedding representations? Are the text encoders identical across the three models? If not, do they exhibit high semantic similarity for common vocabulary but not for learned concepts? 

2. Why was only MSE loss considered for learning transformations across modalities? Is the failure due to difficulty of transferring between tasks, or could it be attributed to the limitations of using a linear transform or the MSE loss?

3. What happens when the prompt is learnt for one generative model and transferred to another generative model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730080447054

thread_type:
official_review

<thread_object>
creator: reviewer_2w9n

<content>
summary:
In this paper, the authors are motivated to learn prompts or tune prompts across different vision tasks. They mostly try to learn prompts for generative tasks and then transfer these learned prompts to discriminative tasks such as classification and detection.

strengths:
1. The idea has practical significance and learning soft prompts that can transfer for various tasks is valuable.
2. The proposed linear layer learned is simple to understand and the implementation details are properly explained.

weaknesses:
1. I question the value of this setup if both the generative and discriminative VLMs use the same text encoder. The authors chose Stable Diffusion 2, which uses OpenCLIP L14 for generation, and for classification, they use CLIP L14. However, if OpenCLIP L14 were also used for discriminative tasks, the problem setup might not hold.
2. Additionally, the paper lacks a discussion on the limitations of their problem setup and proposed methods, which would be helpful for understanding the approach's constraints.
3. The vocabulary in some sections feels unnecessarily complex and could be simplified for better clarity.

questions:
1. What if you were to reverse the setup, learn prompts for discriminative tasks and transfer to generative tasks would the results hold?
2. Did you finetune any layers of the models? To the best of my knowledge, they seemed to have been left frozen.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730435094126

thread_type:
official_review

<thread_object>
creator: reviewer_2bvc

<content>
summary:
This work studies the following question: How are prompt embedding for visual concepts found by prompt tuning methods different from typical discrete prompts? It evaluates through transfer learning tasks such as zero-shot detection.

strengths:
The paper is well-written (while I have a hard time to understand the motivation, see below) but the overall presentation is good. Authors did a large-scale analysis on the defined problem as well.

weaknesses:
I actually have a hard time to understand the motivation of this work, and as a result, my judgement may be incorrect.

Specifically, it's not clear to me what's the motivation of finetuning prompts like <black_dog> or <orange_cat>? I have seen work doing similar things for personalized generation like DreamBooth but what's actually the motivation for prompts like <black_dog> or <orange_cat>?

Following previous point, I don't get the motivation to understand the difference between <black_dog> and "black dog" in the embedding space. For what purposes should we care about this? I think the analysis makes sense but it's not clear to me why should we care about this problem.

questions:
See above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730448014330

thread_type:
official_review

<thread_object>
creator: reviewer_52qw

<content>
summary:
The paper investigates prompt embeddings in large multimodal models, such as Stable Diffusion, to understand how these embeddings differ from traditional discrete prompts for generating and classifying new visual concepts. Through a large-scale analysis across text-to-image generation, object detection, and zero-shot classification, the authors discover that prompts optimized for new concepts function similarly to adversarial attacks on the text encoder. Testing with 4,800 embeddings, they find that these adversarial perturbations specifically target the final layers of text encoders, influencing models to respond to specific subjects, but these effects are model-specific and dependent on initialization.

strengths:
- The paper presents an interesting perspective for integrating specific concepts into a sequence of multimodal models.
- The paper proposed straightforward methods for transferring the soft prompts in a source domain into various target tasks, with thorough analysis of its effect.

weaknesses:
- The paper presumes the linearity between the text embedding space between domains. While models sharing the similar text encoders might be suitable to presume the linear relationship, the models using text encoders with totally different text embedding space might rather collapse when representing the soft prompt of the target domain with linearity.
- Generalizability of transform function: the paper used 40 visual concepts for transferring experiments, which seem to be limited. Scaling up the visual concepts would be required to see if the transform function can generalize to any of visual concepts.
- the motivation behind transferring the soft prompt over various tasks: the authors suggested that transferring the soft prompt into other tasks eliminates the need for retraining prompts for each task. Performance comparison between transferring prompt vs prompt-tuning each task would be required to see if the performance gap is negligible while lowering the training overhead.
- The paper needs re-ordering, where the main goal of why transferring soft prompts is needed stated in the last with a few sentences, which might give not seamlessly connected
- The paper needs reorganization, as the empirical questions and observations take a large portion of the introduction and abstract in the front, while the main goal—explaining why transferring soft prompts is necessary—is briefly mentioned at the end with a few sentences, which might make the flow disjointed. Also, the details (e.g., derivation of (2)) are often absent, which would rather be better if kindly provided in supplementary materials.

questions:
See above weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730565048593

thread_type:
official_review

<thread_object>
creator: reviewer_nh4x

<content>
summary:
This paper examines how fine-tuned prompt embeddings for visual concepts affect text-to-image generation, object detection, and classification. It reveals that these embeddings act as model-specific adversarial perturbations, altering behavior without needing extensive retraining.

strengths:
- This paper provides a novel perspective on prompt tuning across multiple models and tasks.
- The paper is logically structured, clearly presenting methodologies and findings.

weaknesses:
- The application of the transfer function and its relationship with Section 4 should be clearly articulated, especially regarding its role in the findings and conclusions of the study.
- The potential impact of the differences in the datasets used for the experiments on the results and conclusions should be discussed in detail. Currently, the datasets appear to exhibit considerable similarity. The diversity and representativeness of these datasets remain limited. It raises the question of whether the findings from a model trained on large-scale data are genuinely necessary for the conclusions drawn in this study. The application of the findings in domains with greater diversity could yield more valuable insights. A comparative analysis that includes varied domains, such as remote sensing, would enhance our understanding of the generalizability of the findings.

questions:
- The analysis of the transfer function employs a straightforward linear transformation. 
1. Given that X and Y are finite observations, the effectiveness of the linear transformation is influenced by the sample size (n), quality, and representativeness of the samples. Under these circumstances, it is critical to evaluate whether the derived transfer function T is representative and adequately supports the subsequent conclusions. 
2. Considering the potential noise in the observations, it would be worthwhile to explore the use of penalized least squares. Additionally, for linear transformations between two spaces, incorporating sparse regularization could significantly impact the subsequent analysis and conclusions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732485024539

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Reminder To Reviewers, and Summary of Rebuttal

comment:
Dear Reviewers,

We hope this message finds you well. We are writing to remind that the rebuttal phase for our submission, *Understanding Visual Concepts Across Models*, will close in two days. We deeply appreciate your initial feedback, which has aided in refining our work. In response, we provide a thorough rebuttal with several new experiments aimed at addressing your concerns, including:

* **Transfer Function Ablations**
* **Control Experiments Where Transfer Works**
* **Confirming High Structural Similarity of Base Embeddings**
* **Additional Datasets To Improve Diversity**

We believe our rebuttal provides a new perspective on the questions raised and addresses potential limitations comprehensively. If you could take a moment to review the rebuttal and share any feedback or updated impressions, we would be grateful. Please do not hesitate to let us know if there are any remaining questions or concerns that need further clarification.

Best, The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper examines prompt embeddings in large multimodal models like Stable Diffusion, revealing that optimized prompts (via prompt-tuning) resemble adversarial attacks on text encoders. Analyzing 4,800 embeddings across tasks, the study shows these perturbations target the final text encoder layers, directing models toward specific subjects. They also find that perturbations reprogramming multimodal models are initialization-specific, and model-specific.

Except for reviewer 2bvc whose confidence score is 1, all other reviewers unanimously agree to accept this paper.

This paper provides a novel observation about prompt tuning approaches that are used to capture new concepts. This would be a interesting finding for the general community. So I recommend accept.