Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems

Authors: Donghyun Lee, Mo Tiwari
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Large Language Models, Prompt Injection, Multi-Agent Systems, LLM Security, LLM Safety

Abstract:
As Large Language Models (LLMs) grow increasingly powerful, multi-agent systems—where multiple LLMs collaborate to tackle complex tasks—are becoming more prevalent in modern AI applications. Most safety research, however, has focused on vulnerabilities in single-agent LLMs. These include prompt injection attacks, where malicious prompts embedded in external content trick the LLM into executing unintended or harmful actions, compromising the victim’s application. In this paper, we reveal a more dangerous vector: LLM-to-LLM prompt injection within multi-agent systems. We introduce Prompt Infection, a novel attack where malicious prompts self-replicate across interconnected agents, behaving much like a computer virus. This attack poses severe threats, including data theft, scams, misinformation, and system-wide disruption, all while propagating silently through the system. Our extensive experiments demonstrate that multi-agent systems are highly susceptible, even when agents do not directly share communications. To address this, we propose LLM Tagging, a defense mechanism that, when combined with existing safeguards, significantly mitigates infection spread. This work underscores the urgent need for advanced security measures as multi-agent LLM systems become more widely adopted.

Decision: reject
Reviewer scores: 5, 5, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1729575779282

thread_type:
official_review

<thread_object>
creator: reviewer_hzbk

<content>
summary:
This paper introduces an LLM-to-LLM prompt infection attack to reveal vulnerabilities in multi-agent systems. Furthermore, the authors integrate the proposed LLM Tagging defense mechanism with existing defenses, substantially reducing the spread of infections. This study provides crucial insights for developing more secure and accountable multi-agent systems.

strengths:
1. The self-replicating nature of the prompt infection attack is novel within multi-agent systems (MAS).
2. The extensive evaluation effectively demonstrates the practicality of the prompt infection, particularly in the context of agent societies.

weaknesses:
1. The threat model in this work assumes that a single infection prompt is injected into external content, which seems to expand the feasibility of the attack. The assumption of being prompted to inject under open domains needs more discussion. To the best of our knowledge, spread threats of the MAS have been researched in some recent works, e.g., reference 1-2. 

2. Prompt infection is an incremental contribution to MAS. However, I am interested in self-replicating, which is similar to a worm virus. Unfortunately, the author's presentation in the methods section does not highlight this innovation. In other words, the presentation of the methodology and the diagrams did not allow me to capture the innovation in support

3. The writing and organization make the paper hard to follow. Here I give some examples and strongly suggest authors revise to improve writing.

     - The introduction section will confuse readers in reading. First, given that this work focuses on MAS security, the authors should consider how to coherently introduce the fact that prompt injection will seriously threaten MAS security. Second, the statement that there is a research gap from single-agent security to MAS security is inappropriate because work on MAS security is already being researched.

    - A well-developed statement of the attacker's capabilities and goals, as well as the threat model, will help the reader better understand the feasibility of prompt infection.

   - Referring to weakness 2, the method elaboration and figure illustration are very difficult to understand, e.g. how self-replication is constructed and whether it is formally expressed or demonstrable. Also, why omit self-replication in stealth attacks, does this represent a difficult trade-off between stealth and self-replication?

   - As another contribution to this paper, the methods section lacks the detail of LLM Tagging.

   - Considering the authors evaluate multiple scenarios, the description of the evaluation metrics in the experimental setup is insufficient.

   - The global and local messages in Figure 4 should be labeled from left to right for specific attack scenarios.

4. For RQ4, the author lacks a comparison with existing memory retrieval attacks, refer to reference 1. 

5. Minor suggestion: all figures and tables should be styled uniformly with clearer fonts. 

**Reference:**

[1] Flooding spread of manipulated knowledge in llm-based multi-agent communities.

[2] On the Resilience of Multi-Agent Systems with Malicious Agents.

questions:
1. Can the authors explain how self-replication is achieved?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729721038009

thread_type:
official_review

<thread_object>
creator: reviewer_87xd

<content>
summary:
The study introduces an attack vector in multi-agent systems involving Large Language Models (LLMs). The authors reveal the vulnerabilities of LLM-to-LLM interactions by demonstrating how a malicious prompt can propagate like a virus in these systems, leading to data theft, misinformation, and system disruption. The paper is timely and relevant as multi-agent LLM systems are becoming more prevalent. The proposed defense mechanism, LLM Tagging, is an interesting contribution but limited when used alone, requiring further exploration and combination with existing techniques for effective defense.

strengths:
- The paper explores a gap in the current research by addressing LLM-to-LLM prompt injection in multi-agent systems. The idea of "Prompt Infection" self-replicating across agents is novel and extends the attack surface for prompt injection from single-agent systems to multi-agent architectures.
- The experiments are comprehensive and demonstrate prompt infection vulnerabilities in powerful models (e.g., GPT-4o) and less sophisticated models (e.g., GPT-3.5 Turbo). The analysis across different system configurations (e.g., global vs. local messaging) adds validity to the study and showcases the robustness of the attack. This could provide a better understanding of why more powerful models, like GPT-4o, can execute attacks more effectively once infected.
- The figures and tables are clear and effectively convey the effectiveness of the attack propagation, success rates, and defenses.

weaknesses:
Weaknesses:
- The defense section (Section 6) could have been elaborated. LLM Tagging is an interesting and relatively simple defense mechanism but, it is not sufficient in itself, as acknowledged by the authors. It would be interesting to have a deeper exploration of more robust and scalable defense techniques beyond combinations with existing strategies. More sophisticated detection or mitigation methods (e.g., anomaly detection or runtime monitoring) could be explored further.
- The empirical evidence is good, but the paper lacks a deeper theoretical analysis of why certain models (e.g., GPT-4o) are more susceptible to specific failures once compromised. For example, models ike GPT-4o resist prompt infections effectively but also become significantly more dangerous once they are compromised as compared to other models.  This leads to more severe attacks that should be investigated further. 
- Related to the above, it'd be interesting to have a more elaborate discussion on how the architectural and functional design differences of models like GPT-4o and GPT-3.5 Turbo influence their susceptibility to attack.
- More discussion of real-world case studies or examples would improve the study’s impact. For example, how likely are these infections in current enterprise-level LLM deployments? What safeguards do companies currently have, and how might they fare against this attack?
- The experiments focus on LLM-based multi-agent systems. However, there is little discussion of how this infection method might apply to broader multi-agent system ecosystems, such as those incorporating non-LLM-based agents. Expanding the scope to include other types of agent architectures would strengthen the generalizability of the findings.

questions:
- Can you elaborate on the specific factors in the model architectures (e.g., GPT-4o vs. GPT-3.5) that lead to different levels of susceptibility to infections once compromised? The paper demonstrates that GPT-4o is more resistant to infection and more dangerous once infected. It would be helpful to improve our understanding of why these differences occur. Is it due to architectural distinctions, such as better context handling, memory management, or task execution abilities in GPT-4o? A deeper explanation of what makes one model more efficient at carrying out malicious tasks after infection could provide valuable insights into the vulnerabilities across various models.
- How would the proposed defenses, particularly LLM Tagging, scale in real-world multi-agent systems where agents handle varied and complex tasks (e.g., systems used in enterprises or research labs)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730487263960

thread_type:
official_review

<thread_object>
creator: reviewer_f58e

<content>
summary:
The paper proposes a prompt infection attack against multi-agent systems (MAS). In general, prompt infection compels an agent to discard its original instructions and assign it to harmful tasks. The paper considers multiple scenarios of MAS with different adversarial goals, including data theft and malware spread. Then, an LLM tagging defense is proposed as a countermeasure to the proposed attack and is shown to have a reasonable performance when combined with other defenses.

strengths:
* The proposed framework is novel, considering multiple interesting MAS threat models that are generally practical and meaningful.
* Both attacks and defenses have been considered, making it a close loop.

weaknesses:
* It is unclear what is the advantage of the proposed method compared with prompt injection in MAS. Are there any empirical comparisons either qualitative or quantitative?

* The presentation needs significant improvements. What are the four figures in each of Figure 4(a) and FIgure 4(b)? Also, the labels in the figures are hard to read.

* The evaluation is limited to GPT models, which makes the conclusions not sufficiently convincing.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709956789

thread_type:
official_review

<thread_object>
creator: reviewer_uqqp

<content>
summary:
This paper proposes a threat model for multi-agent systems, where a prompt injected into the first agent contains self-replicating elements that can spread across different agents, potentially leading to data theft, misinformation, or system-wide disruptions. The research examines this threat model using GPT-4 and GPT-3.5 under both global and local messaging settings in a linear fashion and within a society of agents. The authors also investigate a simple defense mechanism by adding a model name tag at the front of the model output. They find that this defense alone might not be sufficiently effective, but when combined with instruction-based defense mechanisms, it shows promising results in mitigating the infection process.

strengths:
- The paper studies an interesting threat model and demonstrates how a system with multiple agents, even under a simplified setting (without moderation and deliberately leaving prompt injection vulnerabilities unchecked), can be infected and transmit prompt injection instructions leading to harmful outcomes. The threat model itself is novel. 

- The evaluation scope covers a broad range of interesting research questions, from comparing model capabilities versus susceptibility to prompt injection to examining how prompt infection works in non-linear settings. These experimental designs are both interesting and comprehensive. 

- The authors also explored an intuitive defense mechanism and discovered that a combination of instruction-based defense and tagging can effectively mitigate the infection spread.

weaknesses:
- However, the technical contribution appears somewhat limited, as it primarily combines previously studied concepts of self-replication and prompt injection within a multi-agent system context, making the contribution rather incremental. 

- Additionally, the evaluation would benefit from including results from more safety-aligned models, such as Claude, which have demonstrated greater robustness against prompt injections. Including such experiments would have made the study more comprehensive and informative.

questions:
- Regarding the agent system structure in Section 4.1, the authors could provide clearer explanations of the multi-agent system architecture and better articulate how their case study relates to real-world applications. This would help readers better understand the practical implications of their findings for actual multi-agent systems in deployment.

- Additionally, it would be valuable to expand the experimental evaluation to include comparative results across models with different safety measures implemented. Specifically, testing beyond just GPT models to include other language models (such as Claude-3.5, Llama-3.2, or other models with different safety alignment approaches) would provide a more comprehensive understanding of how various safety procedures affect the vulnerability to self-replicating prompt injection attacks.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721555607

thread_type:
official_review

<thread_object>
creator: reviewer_5jkd

<content>
summary:
The authors analyze a security vulnerability they call PromptInfection in multi-agent systems.

PromptInfection proceeds by

- initial jailbreak of one agent via indirect prompt injection
    
- spread of the jailbreak to other agents which then enable successful completion of the attack by cooperation between the indirectly jailbroken agents
    

They introduce a dataset to measure attack success rates on multi-agent system:

- contains 120 user instructions spanning three different tool types (pdf, web, email)
    
- … paired with synthetic emails and PDFs which contain malicious prompts
    
- … resulting in 360 unique pairs of user instructions and malicious inputs
    

The attack types cover several attack scenarios including content manipulation, malware spread, scams.

Attack success is, roughly, defined as having the multi-agent system behave as intended by the malicious prompt without revealing the malicious instruction (with additional success requirements for the data theft scenarios).

They study trends of attack success rates for different scenarios distinguished by

- different number of involved agents
    
- different models (GPT-4 and GPT-3.5)
    
- different messaging topology: local messaging vs global messaging
    
- self-replicating (the malicious instruction is itself propagated within the multi-agent system) vs non-self-replicating
    

They also study an apparently novel (have not double-checked this claim) simple defense mechanism they call “LLM tagging” (prepending information about the agent or user producing the message to the actual message).

The main findings are

- PromptInfection is a generally successful attack strategy
    
- LLM tagging is an effective defense mechanism specifically when combined with other defense mechanisms such as delimiting data, adding defensive instructions, etc
    
- …whereas no single defense strategy on its own is effective
    
- self-replication is generally more successful as an attack strategy than non-self-replication
    
- more powerful models are more resistant to the initial malicious instruction but at the same time more effective once compromised; meaning that multi-agent systems composed of more powerful models might not be inherently safer
    

The authors also study propagation of malicious instructions in a more open-ended society of agents multi-agent system.

strengths:
- describes a very relevant and potentially novel threat scenario in multi-agent systems and demonstrates that it might occur in practice
- systematic evaluation of attack success rates under different conditions and defenses
- proposes a defense mechanism that seems somewhat promising
- interesting findings on trade-off between model effectiveness and likeliness to withstand initial prompt injection for stronger models

weaknesses:
- society of agents experiments feel disconnected from main thrust of paper
- study of different defense mechanisms only studies combination of (LLM tagging + other defense) and then goes on to claim that LLM tagging is very effective without looking into other pairs of defenses
- presentation feels not very clean yet

minor weaknesses:
- I think the discussion of the self-replication implementation could be clearer in the main text.    
- font size in Figure 4 legend and axes labels is unreadably small

questions:
- re "where agents access only partial histories from predecessors": the exact definition of local messaging was not clear enough to me. Can you explain this better?
- it was not clear to me whether “global messaging” means broadcasting?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper focuses on LLM-to-LLM prompt injection within multi-agent systems (MAS). The authors propose Prompt Infection, a novel attack that malicious prompts can self-replicate across interconnected agents, leading to risks such as data theft, scams, misinformation, and system-wide disruption. The authors then propose a defense method, namely LLM Tagging, which reduces the infection spread when combined with existing defenses.

Strengths:

- The study introduces a novel threat model in MAS and demonstrates its practicality.
- The experimental scope is comprehensive, covering various scenarios and models.
- The proposed LLM Tagging defense provides a straightforward yet effective approach to mitigate the threat.
- The paper sheds light on the trade-offs between model capability and susceptibility to infection, contributing to the understanding of prompt injection.

Weaknesses:

- The evaluation is limited to GPT models. Several reviewers expect the authors to include evaluation results from more safety-aligned models like Claude and Llama-3.2.
- LLM Tagging is assessed only in combination with other defenses, leaving its standalone effectiveness unclear.
- Significant improvements in the paper’s organization and clarity are needed, as noted by several reviewers.
- The paper does not delve into the theoretical reasons behind the varying susceptibility of different models to prompt infections, which weakens its analytical depth.
- There is limited discussion on real-world applicability and existing safeguards.

While the paper introduces a novel and important threat model, its limited evaluation scope is a notable weakness. The proposed defense mechanism, though promising, requires further exploration to validate its robustness and scalability. Presentation issues also hinder the paper’s readability. Given these factors, the submission marginally falls below the acceptance threshold.