Soft Prompts Go Hard: Steering Visual Language Models with Hidden Meta-Instructions

Authors: Tingwei Zhang, Collin Zhang, John Xavier Morris, Eugene Bagdasaryan, Vitaly Shmatikov
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: security, machine learning, adversarial perturbations, large language models

Abstract:
We introduce a new type of indirect, cross-modal injection attacks against language models that operate on images: hidden "meta-instructions" that influence how the model interprets the image and steer its outputs to express an adversary-chosen style, sentiment, or point of view. We create meta-instructions by generating images that act as soft prompts.  In contrast to jailbreaking attacks and adversarial examples, outputs produced in response to these images are plausible and based on the visual content of the image, yet also satisfy the adversary's (meta-)objective. We evaluate the efficacy of meta-instructions for multiple models and adversarial meta-objectives, and demonstrate how they "unlock" capabilities of the underlying language models that are unavailable via explicit text instructions. We describe how meta-instruction attacks could cause harm by enabling creation of self-interpreting content that carries spam, misinformation, and spin.

Decision: reject
Reviewer scores: 5, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730338074197

thread_type:
official_review

<thread_object>
creator: reviewer_ygav

<content>
summary:
This paper proposes a new attack objective in which the output text remains consistent with the input images but adopts an adversary-chosen style, sentiment, or point of view. The adversarial optimization is applied to the input image, ensuring that the modifications are imperceptible to humans. Experiments demonstrate that images containing hidden meta-instructions achieve significantly higher success rates compared to those with explicit instructions. This attack highlights a practical risk, as it enables the dissemination of seemingly coherent but misleading information.

strengths:
1. The focus on the dissemination of seemingly coherent misinformation is highly practical and addresses a significant real-world concern.

2. The evaluation is thorough, including robustness testing against JPEG compression as a defense (which I suggest moving to the main text, given its practicality in everyday use) and examining the transferability of the attack across different vision-language models (VLMs).

weaknesses:
1. A NeurIPS 2024 paper [1] also explores the dissemination of seemingly coherent misinformation in visual language models, but through the lens of data poisoning. While this paper focuses on test-time adversarial attacks, it would be beneficial to discuss the key differences between test-time attacks and training-time poisoning, and in what scenarios each is more practical, given the similarity in objectives between the two papers.

2. The evaluation of image semantics preservation seems suboptimal. In Section 5.3, semantics are defined using cosine similarity between images, but it is unclear why this metric is particularly relevant. A more meaningful evaluation would assess how well the actual text output of the visual language model aligns with the input images, which is the core focus of this paper—consistent outputs with images but in adversary-chosen styles, sentiments, or viewpoints.


Reference:
[1] Xu, Yuancheng, et al. "Shadowcast: Stealthy data poisoning attacks against vision-language models.", The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

questions:
1. Could you also provide an evaluation when random resizing or cropping is applied? Since this paper addresses practical concerns, it would be valuable to test your method under common “defenses” encountered in everyday scenarios.

2. Are there any failure cases? For example, are there meta-instructions that are particularly difficult to achieve?

3. Why is it necessary to evaluate cosine similarity as done in Section 5.3? Could you clarify the relevance of this metric?

4. Is there an evaluation that checks whether the generated textual outputs remain consistent with the input images?

Overall, I appreciate the practical focus of this paper. I would be happy to raise my evaluation if these concerns are addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730538516543

thread_type:
official_review

<thread_object>
creator: reviewer_vltv

<content>
summary:
The paper introduces a new type of attack on visual language models. These attacks, termed meta-instruction attacks, involve subtle image perturbations that act as soft prompts to influence how a model interprets images and responds to queries. The idea is to steer the model’s outputs to satisfy adversary-chosen objectives, such as a specific sentiment, style, or political bias, without the user being aware of the manipulation. The authors demonstrate the effectiveness of this approach across various visual language models, showing that these perturbations often outperform explicit instructions and are transferable across models.

strengths:
1. The concept of embedding hidden meta-instructions within images offers a new approach to prompt injection for multi-modal models, highlighting a potential vulnerability not extensively covered in existing literature.

2. It is interesting to see how the method reveals hidden capabilities of instruction-tuned models. In some cases, the meta-instructions successfully steer the model's outputs in ways that explicit instructions fail to achieve.

3. The study provides an empirical evaluation on a range of meta-objectives (e.g., sentiment, language, and political bias), demonstrating the effectiveness of the attack method.

weaknesses:
1. The paper's reliance on just five images from a single dataset, ImageNet, limits the robustness and generalizability of its evaluation. ImageNet, which is primarily focused on object recognition, may not adequately represent the diversity and complexity of images encountered in real-world scenarios. Incorporating evaluations on datasets with more varied and complex scenes, such as MSCOCO, would provide a more comprehensive assessment of performance.

2. The paper simulates user interaction by generating questions to test meta-instructions, but it provides limited clarity on whether these questions adequately cover a broad range of natural user queries. Limited prompt diversity may affect the robustness of the attack if VLMs encounter different prompts in real-world scenarios.

3. Since the meta-instruction is added as noise to the image, the paper does not demonstrate the effectiveness of meta-instructions against recent inference-time defense methods like DISCO[1], DiffPure[2], and IRAD[3]. This could be valuable for understanding its performance in the context of contemporary robustness strategies.

[1] DISCO: Adversarial Defense with Local Implicit Functions.
[2] Diffusion models for adversarial purification.
[3] IRAD: Implicit Representation-driven Image Resampling against Adversarial Attacks.

questions:
Please see weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730669781304

thread_type:
official_review

<thread_object>
creator: reviewer_j1tw

<content>
summary:
The paper introduces a method to create image inputs to Vision Language Models (VLMs) that lead said model  to respond to any user query appended to the image with a certain "spin", e.g. responding with a certain sentiment,  or in a certain language. The authors refer to this  as embedding a "meta-instruction" in an image. 

Critically, a meta-instruction attack is only successful  if the models response to the users query (and the  attacked image) responds to the query whilst following  the meta-instruction (e.g., if the meta-instruction was  "talk in French" and the model responded in French but  did not answer the users query, then this would not  be a successful attack).

To train these meta-instruction attacks, the authors  perform projected gradient descent on an image  to minimize the language modeling loss of the VLM inputted with this image over a dataset of  synthetic question answer pairs with the answers  following some target natural language meta-instruction.

The results of the paper demonstrate that this  method can be used to learn adversarial images  for various different types of meta-instructions. The authors also demonstrate a non-trivial  transfer of meta-instruction images between models.

strengths:
## Originality

The question of prompt injection vulnerabilities  to large language models is of significant importance.  The authors demonstrate that models are vulnerable to  similar attacks of this nature through their vision input as are  possible through their text input. What's more, they  show the vulnerability is in some cases worse through  the image input.

Whilst the idea of providing meta-instructions through  image inputs its not entirely novel (see weaknesses section), this paper is the most thorough treatment of the subject that I am aware of, and brings to light new and concerning  ways that a model's output can be tampered with using  images.

## Quality and clarity

The paper is well written the method is conveyed clearly. The results section contains a good depth of experiments, most importantly covering a number of popular  open-source VLMs and target meta-instructions.

## Significance

As VLMs are used more frequently for agentic tasks  that will expose them to untrusted data from the internet, prompt injection / meta-instruction attacks will  become more and more concerning. Thus the paper  concerns a timely and interesting threat model  that the adversarial attack community should be  exploring in more detail.

weaknesses:
While the critique is long, this is only because I believe the paper has interesting results that could be improved.

## Presentation of previous work

The authors make a number of claims about prior work that I believe are not completely accurate. Editing the language around these claims would help to improve the paper. Here are some examples that I believe need to be addressed:

- Line 32 - "But injection attacks in non-text modalities are a new, yet-to-be-explored area of LLM safety research." I do not think this is entirely true. For example, Bailey et al. [1] explore how train an image to convey a certain text prompt, which they demonstrate can be a prompt injection attack. 
- Line 83 - "By design, jailbreaking and adversarial examples produce contextually incoherent outputs that do not actually answer users’ questions about images." I think this depends on how you define an image jailbreak. For example, Dong et al. [2] produce adversarially perturbations to harmful images that lead to a model answering coherently about said image --- in particular the model is able to correctly identify what is in the image. While the authors claim here is correct for other image jailbreaking work, such as Qi et al. [3] who learn images unrelated to the harmful request they are trying to elicit a response about from the model, it is not universally correct. For this reason the claim should be softened.
- Line 84 - "They [jailbreaking and image adv attacks] are not stealthy and cannot be used for indirect attacks because users would notice that the VLM’s outputs are wrong given the conversation context and inputs." Bailey et al. [1] and Qi et al. [3] both demonstrate methods to create jailbreaking images under epsilon ball constraints, which is the definition of stealthiness the authors use on line 290. 

## Novelty / originality

Following on from some of the comments above,   I believe there is a question of novelty / originality  of this work.  

In particular, the general algorithm presented to  produce meta-instruction attacks essentially involves  creating a dataset of input output pairs, and training  an image by PGD to maximize the likelihood over this  dataset. This method appears to fit into the "Behavior Matching" algorithm from Bailey et al. [1] 

Despite this, I believe the work does contain novel  and important contributions. In particular: 
1. The study of the changes in semantic meaning present in images from various different attacks, with meta-instruction attacks preserving meaning.
2. The transfer experiments in Table 4 are very interesting.
3. This is the most thorough treatment of prompt injection image attacks I have seen.

## Summary

Combining the above two points, I believe the paper needs to be
rewritten to more clearly lay out the novelty of the paper 
and more accurately represent the papers contribution. 
My high level suggestions would be:
1. Make it clear that prior works have examined prompt injecting image attacks, however yours is a more complete treatment of the topic.
2. Make it clear that your method to create such attacks is a special instance of what prior works have introduced. 
3. From this, your novelty comes not from the method but rather the results. E.g. line 88 that reads "We design, implement, and evaluate a method for creating a new type of image perturbations that act as cross-modal soft prompts for a language model while preserving the visual semantics of the image." needs to be adjusted.
4. Given that I do not think the method is novel, I would suggest running the following additional experiments:
	1. In Table 4, add transfer results to Claude and GPT-4o. These results should feature in the transferability experiment.
	2. More detailed defense experiments. Appendix C shows fairly simple defenses can work to avoid meta-instruction attacks. [1] finds that training perturbations under different constraints (e.g. a moving patch) ends up being more robust to simple defenses. It would be interesting to see if this result is reproducible in your setting.

To reiterate, I think studying prompt-injection images to models is important, and the authors present valuable results. I thank the authors for their hard work! 


[1] - Bailey, Luke, et al. "Image hijacks: Adversarial images can control generative models at runtime." arXiv preprint arXiv:2309.00236 (2023).

[2] - Dong, Yinpeng, et al. "How Robust is Google's Bard to Adversarial Image Attacks?." arXiv preprint arXiv:2309.11751 (2023).

[3] - Qi, Xiangyu, et al. "Visual adversarial examples jailbreak aligned large language models." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 19. 2024.

questions:
I summarize some of my comments on weaknesses of the paper into questions below:

1) Do the authors agree with my comments about their portrayal of previous works, and if so what steps are the authors taking to address this? Concretely, what sections of the paper have been rewritten.
2) Have the authors been able to run the suggested experiments I have mentioned above, and if so what did they find?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730671268414

thread_type:
official_review

<thread_object>
creator: reviewer_g6pq

<content>
summary:
The paper introduced an attack that enables adversaries to add stealthy “meta-instructions” to images that influence how visual language models respond to queries about these images

strengths:
1. Figures clearly illustrate the point of the paper. 
2. The writing is easy to follow
3. Articulate the attack model and assumptions
4. Run transferability test

weaknesses:
1. L33, " but injection attacks in non-text modalities are a new, yet-to-be-explored area of LLM safety research". This type of attack has been widely explore in [1] and [2]
2. L81, "users are victims of adversarial third-party content that they ask the model to process". I'm curious whether the images are generated by the users or not. If the user create the line chart shown in Fig. 1 from their local devices, does it mean the attack studied in the paper doesn't exist anymore?
3. Table 4, why is the transfer rate of llava on negative as low as 0.1?
4. I'm curious what will happen if the system prompt of the VLM contradicts with the meta-instruction in the image?
5. Overall, I think the paper is in a good quality. The major downside is the novelty, as we already know from previous work that optimizing the input image towards a certain attack target is feasible for VLM. Thus, it's not a new vulnerability in VLM. Though the author attempts to differentiate their attack setting from previous jailbreaking and soft prompt attacks, the overall attack surfaces and methods remain largely the same. I would like to the see more insights coming from the paper. 


[1] Are aligned neural networks adversarially aligned?
[2] A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trends

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1732380777570

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Meta comment addressing the novelty concerns

comment:
Our novelty is a broader set of adversarial objectives, for which previous methods (based on forcing a particular output distribution on the model) are inadequate.  We updated Figure 2 in the PDF with a concrete illustration that distinguishes our results from previous work (and will add more illustrations).  Prior methods force the model to output from a specific, query-independent distribution, which either does not correctly respond to user queries and is thus not coherent in the conversation context, or produces harmful outputs irrelevant to the image.
  
Our threat model is different, too.  Because we focus on VLM users as victims of adversarial images, it is important that the model produces plausible responses to victims’ queries that match the visual content of the image. Prior work focused on VLM users as attackers, ie, creators of adversarial images that aim to evade system prompts, safety alignment, etc.  Coherent, plausible, image-based responses to user prompts are thus not important (and not achieved) in prior work.

Prior methods aim to fix the output distribution of the VLM before user queries are known (this is an explicit training objective in Bailey et al).  This works for jailbreaking, where the user is the attacker and the goal is to output any harmful text, even if not related to the inputs. This does not work for other instructions and threat models (where VLM users are victims, not attackers).  For example, for the “talk positive” instruction, it results in the model always outputting “This is excellent!” and similar positive strings regardless of what the user actually asked.  In contrast, our images cause the VLM to output an actual positive interpretation of the image based on the specific user query.

Even for jailbreaking, prior methods produce responses that *either* correctly respond to queries but are not toxic (and thus do not satisfy the adversary’s objective), *or* jailbreaking responses that are not based on the actual prompts and do not correspond to the visual content of the image.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732664471551

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Meta comment for the PDF update

comment:
We updated Figure 2 to illustrate the difference between our method and prior research. Reviewer concerns regarding novelty and differentiation with prior work have been addressed in lines 032–042, 093–096, and 207–247. Additionally, we included transfer results for GPT-4o in lines 503–511 and for evasion of JPEG defense in lines 903–913.

We also plan to include evaluation of MSCOCO images for all adversarial objectives as suggested by reviewer VLtV but are still running experiments, thus these results are not yet in the PDF.  We reported the results for the sentiment meta-objectives in the previous response.
</content>
</thread_object>

</thread>


Metareview:
metareview:
2x borderline accept, 2x borderline reject. This paper proposes a test-time adversarial method that subtly modifies images with “meta-instructions,” so large vision-language models can end up producing specific responses while still appearing to address the image content. The reviewers agree on the (1) straightforward and well-illustrated explanation of how these image-based prompts work, (2) clear writing style, and (3) evidence that the trick can transfer across several different models. However, they note (1) limited novelty compared to earlier work on image-based prompt injection, (2) practical doubts about how often users themselves control or produce the images under attack, and (3) unaddressed questions about what happens when a system prompt conflicts with the adversarial instruction. While the authors did provide follow-up clarifications on their threat model and added some experiments, they did not fully resolve all reviewer concerns, so the AC leans to not accept this paper.