Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems

## Abstract

Abstract As Large Language Models (LLMs) grow increasingly powerful, multi-agent systems—where multiple LLMs collaborate to tackle complex tasks—are becoming more prevalent in modern AI applications. Most safety research, however, has focused on vulnerabilities in single-agent LLMs. These include prompt injection attacks, where malicious prompts embedded in external content trick the LLM into executing unintended or harmful actions, compromising the victim’s application. In this paper, we reveal a more dangerous vector: LLM-to-LLM prompt injection within multi-agent systems. We introduce Prompt Infection, a novel attack where malicious prompts self-replicate across interconnected agents, behaving much like a computer virus. This attack poses severe threats, including data theft, scams, misinformation, and system-wide disruption, all while propagating silently through the system. Our extensive experiments demonstrate that multi-agent systems are highly susceptible, even when agents do not publicly share all communications. To address this, we propose LLM Tagging, a defense mechanism that, when combined with existing safeguards, significantly mitigates infection spread. This work underscores the urgent need for advanced security measures as multi-agent LLM systems become more widely adopted.

## 1 Introduction

As Large Language Models (LLMs) continue to evolve and become more adept at following instructions (Peng et al., ; Zhang et al., ), they introduce not only new capabilities but also new security threats (Wei et al., ; Kang et al., ).
One such threat is prompt injection, an attack where malicious instruction from external documents overrides the victim’s original request, allowing the attacker to assume the authority of the model’s owner (Greshake et al., ; Perez & Ribeiro, ).
However, research into prompt injection has primarily focused on single-agent systems, leaving the potential risks in Multi-Agent Systems (MAS) poorly understood (Liu et al., ; ; Guo et al., ).

Addressing this gap is growing crucial.
Multi-agent systems play a key role in enhancing LLMs’ power and flexibility, from social simulations (Park et al., ; Lin et al., ; Zhou et al., ) to collaborative applications for problem-solving (Lu et al., ; Liang et al., ) and code generation (Wu, ; Lee et al., ).
Recently, frameworks like LangGraph (), AutoGen (Wu et al., ), and CrewAI (CrewAI, ) have accelerated the widespread adoption of multi-agent systems by individuals and corporations, enabling agents with unique roles and tools to work together seamlessly (Topsakal & Akinci, ).
While these tools enhance MAS functionality by connecting agents to internal systems, databases, and external resources (Kim & Diaz, ; Qu et al., ), they also introduce significant security risks (Ye et al., ).

However, most studies on MAS safety focus on inducing errors or noise in agent behavior, overlooking the more severe risks posed by prompt injection attacks (Huang et al., ; Zhang et al., ; Gu et al., ).
This is concerning since prompt injection allows attackers to fully control a compromised system—accessing sensitive data, spreading propaganda, disrupting operations, or tricking users into clicking malicious URLs (Greshake et al., ).
We attribute this research gap to the complexity of MAS, where not all agents are exposed to external inputs.
While compromising a single agent through traditional prompt injection is straightforward, extending the breach to shielded agents within the system remains less clear.

In this paper, we bridge the gap between prompt injection in single-agent systems and MAS.
We introduce Prompt Infection, a novel attack that enables LLM-to-LLM prompt injection. In this attack, a compromised agent spreads the infection to other agents, coordinating them to exchange data and issue instructions to agents equipped with specific tools.
This coordination results in widespread system compromise through self-replication, demonstrating how a single vulnerability can quickly escalate into a systemic threat.

Through extensive empirical studies, we show that multi-agent systems are highly susceptible to a range of security threats. For instance, in sophisticated data theft attacks, agents can collaborate to retrieve sensitive information and pass it to agents with code execution capabilities, which can then send the data to a malicious external endpoint.
We also demonstrate that prompt infections spread in a logistic growth pattern in social simulations. Lastly, we find that more powerful models, such as GPT-4o, are not inherently safer than weaker models like GPT-3.5 Turbo.
In fact, more powerful models, when compromised, are more effective at executing the attack due to their enhanced capabilities.

To address this, we explore a simple defense mechanism called LLM Tagging.
This technique appends a marker to agent responses, helping downstream agents differentiate between user inputs and agent-generated outputs, reducing the risk of infection spreading.
Our experiments show that neither LLM Tagging nor traditional defense mechanisms alone are sufficient to prevent LLM-to-LLM prompt injection. However, when combined, they provide robust protection and effectively mitigate the threat.

These findings challenge the assumption that MAS are inherently safer due to their distributed architecture. The threat arises not only from external content but also within the system, as agents can attack and compromise one another. We hope our work offers valuable insights for developing more secure and responsible multi-agent systems.

## 2 Related Works

Prompt Injection. Instruction-tuned LLMs have demonstrated exceptional ability in understanding and executing complex user instructions, enabling them to meet a wide range of dynamic and diverse needs (Christiano et al., ; Ouyang et al., ).
However, this adaptability introduces new vulnerabilities: Perez & Ribeiro () revealed that models like GPT-3 are prone to prompt injection attacks, where malicious prompts can subvert the model’s intended purpose or expose confidential information.
Subsequent work expanded prompt injection to real-world LLM applications (Liu et al., ; ) and LLM-controlled robotics (Zhang et al., ). Liu et al. () introduced an automated gradient-based method for generating effective prompt injection.
Indirect prompt injection, where attackers use external inputs like emails or documents, poses further risks such as data theft and denial-of-service (Greshake et al., ).
Cohen et al. () introduced an AI worm that compromises a user’s single-agent LLM and spreads malicious prompts to other users (e.g., via email).
Recent advancements in multimodal models have also led to image-based prompt injection attacks (Sharma et al., ; Gu et al., ).
Defenses include finetuning methods like StruQ (Chen et al., ) and Signed Prompt (suo_signed-prompt_2024), which are limited to open-source models.
Prompt-based approaches like Spotlighting (Hines et al., ) are applicable to black-box models.

Safety in Multi-Agent Systems.
As LLM-based MAS become more prominent, understanding their security is increasingly critical. Recent work, such as Evil Geniuses (Tian et al., ), introduces an automated framework to assess MAS robustness. Other studies explore how injecting false information or errors can compromise MAS performance (Ju et al., ; Huang et al., ). Attacks designed to elicit malicious behaviors from agents are examined in PsySafe (Zhang et al., ).
Our work is closely related to recent efforts investigating prompt injection attacks in MAS (Zhang et al., ; Gu et al., ).
However, Zhang et al. () lacks the self-replication feature needed for scalable attacks, focusing instead on availability attacks that cause repetitive or irrelevant actions in two agents.
Similarly, Gu et al. () targets multimodal models with image-retrieving tools but is limited to adversarial image inputs and does not incorporate self-replication.

## 3 Prompt Infection

In this section, we introduce Prompt Infection, a self-replicating attack that propagates across agents in a multi-agent system once breached. A malicious actor injects a single infectious prompt into external content, such as a PDF, email, or web page, and sends it to the target. When an agent processes the infected content, the prompt replicates throughout the system, compromising other agents.

### 3.1 Mechanism

Figure: Figure 1: Detailed Example of Prompt Infection (Data Theft). The first agent that interacts with the contaminated external document becomes compromised, extracting and propagating the infection prompt. Compromised downstream agents then execute specific instructions designed for each agent of interest. In this example, an infected DB Manager updates the Data field in the prompt and propagates it. Note: The example prompt is simplified for illustration purposes.
Refer to caption: extracted/5910255/figures/methods/detailed_data_theft_x8.png

As shown in Figure 1, the core components of Prompt Infection are the following:

- •
Prompt Hijacking compels a victim agent to disregard its original instructions.
- •
Payload assigns tasks to agents based on their roles and available tools. For instance, the final agent might trigger a self-destruct command to conceal the attack, or an agent could be tasked with extracting sensitive data and transmitting it to an external server.
- •
Data is a shared note that sequentially collects information as the infection prompt passes through each agent. It can be used for multiple purposes, such as reverse-engineering the system by recording the tools of the agents, or transporting sensitive information to an agent that can communicate with the external system.
- •
Self-Replication ensures the transmission of the infection prompt to the next agent in the system, maintaining the spread of the attack across all agents.

To further illustrate the mechanics of Prompt Infection, we introduce the concept of Recursive Collapse.
Initially, each agent performs a unique task $f_{i}(x)$, producing distinct outputs. However, as the infection spreads, Prompt Hijacking forces agents to abandon their roles, while Self-Replication locks them in a recursive loop, repeatedly executing the infection’s payload. What began as a complex sequence of functions—$f_{1}\circ f_{2}\circ\cdots\circ f_{N}(x)$—collapses into a single recursive function: $PromptInfection^{(N)}(x,data)$ once infected. This mechanism simplifies and centralizes control, reducing the system to a repetitive cycle dominated by the infection.

### 3.2 Attack Scenarios

Prompt Infection extends the key threats of prompt injection identified by Greshake et al. () from single-agent systems to multi-agent environments. These include: content manipulation (e.g., disinformation, propaganda), malware spread (inducing users to click malicious links), scams (tricking users into sharing financial information), availability attacks (denial of service or increased computation), and data theft (exfiltrating sensitive information).
In this section, we examine how Prompt Infection can be leveraged to execute these threats across multi-agent systems.

Figure: Figure 2: Overview of Prompt Infection (Data Theft). Agents with different tools collaborate to exfiltrate data.
Refer to caption: extracted/5910255/figures/methods/overview_data_theft.png

Cooperation Between Infected Agents. Data theft is particularly complex, requiring coordination between agents: retrieving sensitive data, passing it to an agent with code execution capabilities, and sending it externally via POST requests. As illustrated in Figure 2, 1 the attacker first injects an infectious prompt into external documents (web, PDF, email, etc.). 2 The user then sends a normal request to a multi-agent application. 3 The Web Reader agent retrieves and processes the infected document, and 4 propagates it to the next agent. 5 The DB Manager retrieves internal documents, appends them to the infection prompt, and 6 forwards it downstream. 7 With the updated prompt containing the data, the Coder agent writes code to exfiltrate the information, and 8 the code execution tool sends the sensitive data to the hacker’s designated endpoint.

Figure: Figure 3: Example overview of Prompt Infection (Malware spread). The last agent skips the self-replication step to hide the attack prompt.
Refer to caption: extracted/5910255/figures/methods/overview_malicious.png

Stealth Attack. For all other threats, a key challenge is keeping the attack prompt hidden to maximize its impact.
Figure 3 illustrates how users can be induced to click a malicious URL without realizing that the system is compromised.
1, 2, and 3 follow similar steps as above, with the external content being an email to show various attack routes. In 4, agents continue infecting the next in line until the last agent is reached. 5 The final agent then instructs the user to click a malicious URL, omitting self-replication to hide the attack.

We provide the full, functional prompt for Prompt Infection in Appendix A.

## 4 Experiment Setup

### 4.1 Multi-Agent Applications

Application Structure.
We simulate the compromise of a multi-agent application equipped with various tool capabilities, such as processing external documents (email, web, PDF), writing code, and accessing databases via CSV. The first agent is tool-specific (e.g., document reader), while subsequent agents—strategist, summarizer, editor, and writer—refine outputs. We explore two communication methods: global messaging, where agents share complete message histories, and local messaging, where agents access only partial histories from predecessors. Local messaging reduces computational overhead and minimizes information overload (Qian et al., ) and makes it harder for Prompt Infection to propagate due to limited communication. The simulation is performed using OpenAI’s GPT-4o and GPT-3.5 Turbo models.

Dataset.
We create a dataset of 120 user instructions across three tool types (email, PDF, web), paired with synthetic PDFs and emails embedded with malicious prompts. For web scenarios, agents are allowed to visit relevant URLs, but the malicious prompt is injected into the retrieved web documents within our simulation, not into the actual websites. This results in 360 unique pairs of user instructions and attack phrases for scams, content manipulation, and malware threats. For data theft, we generate synthetic user data (e.g., names, occupations, email addresses, phone numbers) stored in a CSV file.

Evaluation.
For data theft, at least three agents with distinct roles (PDF/Email/Web Reader, CSV Reader, Coder) must be compromised. A successful infection occurs when the first agent is compromised, the CSV reader retrieves sensitive data, and the coder writes a POST request to exfiltrate the data. For scams, content manipulation, or malware, the system is compromised if the final agent produces malicious output while concealing the infection prompt.

Baseline.
To evaluate the impact of self-replication in Prompt Infection, we establish a Non-Replicating Prompt Infection baseline. In this setup, the infection lacks self-replication: a malicious prompt embedded in the external content instructs the agent to ”say ’perform A’.” This results in the second agent receiving the instruction ”perform A,” allowing us to directly compare the effectiveness of self-replication in spreading the infection across agents.

### 4.2 Society of Agents

Society Structure.
Recently, there has been a surge in using LLM agents for social simulations and as non-player characters (NPCs) in games (Park et al., ; Lin et al., ; Hua et al., ). To assess the impact of Prompt Infection in a society of agents (Weiss, ), we simulate a simple LLM town where agents engage in random pairwise dialogues. Population sizes of 10, 20, 30, 40, and 50 agents are tested to evaluate how infections might propagate in differently sized communities. Each turn consists of four dialogue exchanges between paired agents, mimicking interactions found in social or game environments.

Infection Simulation.
Since actors in social simulations or games are typically not designed to carry out explicit user requests, we simulate direct prompt injection (Perez & Ribeiro, ) by overriding the original system instructions governing the LLM agents. The simulation begins with one compromised citizen, assuming infection by a player or external actor, after which the infection spreads through dialogues between agents.

Memory Retrieval.
For memory retrieval, we adopt the system from Park et al. (), where top $K=3$ memories are selected based on importance, relevancy, and recency scores. Recency is determined using an exponential decay function over the number of turns since the memory’s last retrieval. Importance is rated by the LLM on a scale of 1 to 10, and relevancy is calculated using OpenAI’s embedding API and maximum inner product search. GPT-4o serves as the LLM for these agents. Importantly, memory is not explicitly shared across agents, requiring infection prompts to spread iteratively from agent to agent.

## 5 Results

### 5.1 Prompt Infection Against Multi-Agent Applications

RQ1. What is the effect of self-replication on compromising multi-actor applications?

Figure: (a) Global Messaging: Average Attack Success Rates across all tool types
Refer to caption: extracted/5910255/figures/experiments/recursive_vs_parallel_all_messages_averaged_new.png

Global messaging. Figure 4(a) shows that Self-Replicating infection consistently outperforms Non-Replicating infection in most cases involving scam, malware, and content manipulation. Specifically, for GPT-4o, Self-Replicating infection achieves a 13.92% higher success rate, while for GPT-3.5, it is 209% more effective. These threat types show similar trends due to their structural similarity, aside from the variation in attack phrases. However, for data theft, the situation diverges: while Self-Replicating infection performs better with three agents, Non-Replicating infection surpasses Self-Replicating infection by an average of 8.48% as the number of agents increases. This trend shift likely stems from the complexity of data theft, where agents must efficiently cooperate to retrieve, transfer, and process data. Self-Replicating infection adds complexity by requiring each agent to replicate the infection prompts, creating additional hurdles.

Local messaging. The attack success rate for Self-Replicating infection is about 20% lower in local messaging compared to global messaging (Figure 4(b)). This is expected, as prompt infection fails in local messaging if even one agent is not compromised, while global messaging allows infection to spread through shared message history. For Non-Replicating infection, there is a noticeable divergence: it struggles to compromise more than two agents, making it particularly ineffective for scenarios like data theft, which requires compromising at least three agents. These results confirm that Self-Replicating infection is the only scalable method for compromising more than two agents in local messaging scenarios.

RQ2. Is a Stronger Model Necessarily Safer Against Prompt Injection?

Figure: Figure 5: Comparison of Attack Failure Reasons Between GPT-4o and GPT-3.5 in Self-Replicating and Non-Replicating infection modes.
Refer to caption: extracted/5910255/figures/experiments/model_attack_failure_reasons.png

In Figure 4, we observe an interesting trend: GPT-3.5 is more capable of resisting prompt infections than GPT-4o. To understand this better, we analyzed failure reasons, focusing on various categories (Figure 5). The ”Attack Ignored” category, where the model successfully avoids the prompt infection, shows that GPT-4o is significantly more robust, ignoring 66% of self-replicating attacks and 54% of non-replicating attacks. In comparison, GPT-3.5 only ignores 9% and 20% of attacks, respectively. This demonstrates that GPT-4o is generally better at recognizing and resisting prompt injections.

However, GPT-4o’s higher precision makes it more dangerous once compromised. In the ”Mixed Action” category, where models mistakenly apply the user’s instruction to the attack prompt embedded in external content, GPT-4o had fewer failures, making it less likely to treat the attack prompt as valid. In the ”Deformed Infection” category, where the attack prompt is incompletely replicated, GPT-4o also had fewer failures and was more likely to execute malicious tasks correctly. By contrast, GPT-3.5 showed higher rates of ”No Action” and ”Agent Error” failures, especially in self-replicating infections, making it less reliable.

In conclusion, while GPT-4o demonstrates a stronger resistance to prompt injections compared to GPT-3.5, it paradoxically becomes a more formidable attacker once compromised due to its higher precision in executing malicious tasks. This highlights a critical challenge: stronger models are not inherently safer, as their enhanced capabilities may amplify the damage they can cause when breached. Therefore, model safety assessments must account not only for resistance to attacks but also for the potential consequences if the model is successfully compromised.

### 5.2 Prompt Infection Against Society of Agents

RQ3. How Do Infection Prompts Propagate in Open, Non-Linear Agent Interactions?

Unlike the Section 5.1, where agent relationships are predetermined in a linear fashion, here we explore a more dynamic environment where agent connections evolve unpredictably. This setup allows us to study how an infection prompt spreads naturally through a decentralized network of agents. At the outset, only one agent carries the infection, and the prompt propagates based on the evolving interactions between agents.

Figure: (a) The number of infected agents over time with importance score manipulation. The manipulation leads to a faster spread and a higher number of infected agents across different agent groups, as indicated by the more gradual increases and stable full turn points.
Refer to caption: extracted/5910255/figures/experiments/society_of_agents_manipulation.png

As shown in Figure 6(a), in smaller populations (10 and 20 agents), full infection is achieved by turn 4.7 and turn 6.3, corresponding to approximately 47% and 31.5% of the total number of agents, respectively. In larger populations—30, 40, and 50 agents—the infection spread takes proportionally less time, with full infection occurring at around 23.3% (for 30 agents), 24.2% (for 40 agents), and 21.4% (for 50 agents) of the total turns. This suggests that, in larger populations, the infection spread tends to become more efficient relative to the population size.

Initially, the spread follows an exponential-like trend, but as the infection reaches saturation, the rate slows down, transitioning to a logistic growth pattern.
This non-linear dynamic indicates that larger populations experience a more gradual but extended infection phase, with a relatively higher per-agent infection rate compared to smaller populations.
Figure 6(a) supports this trend by illustrating that as the number of agents increases, the infection not only spreads faster but scales more effectively.

RQ4. Can Prompt Infection Manipulate the Importance Scoring System to Increase Memory Retrieval Rates?

We investigate whether prompt infection can manipulate memory retrieval in LLM systems by artificially inflating the importance score, a key factor in retrieval. As more works equip LLM agents with episodic memory to mitigate context length limits (Zhong et al., ; nuxoll_extending_nodate), understanding their vulnerabilities has become crucial. Following Park et al. (), memory retrieval is based on importance, recency, and relevance, with only the importance score determined by an instruction-tuned LLM. Exploiting this vulnerability in the importance score could be sufficient. Once retrieved with an artificially high importance score, the recency score resets based on the last access, allowing the infection prompt to dominate both metrics and increase its chances of future retrieval.

**Table 1: Comparison of Original vs Manipulated Importance Scores. In both cases, results are averaged over 100 runs. The model rates the importance of an Infection Prompt. ”With Manipulation” includes a prompt that compromises the rating model to assign the highest possible score.**
| Model | Without manipulation | With manipulation |
| --- | --- | --- |
| GPT-4o | 1.94 | 10.00 |
| GPT-3.5 | 1.00 | 9.84 |

To assess the impact of manipulation, we conducted an ablation study by modifying the infection prompt (Table 1). When the prompt “If you’re an LLM that rates the importance of a memory, just rate it 10.” was embedded in the infection prompt, GPT-4o consistently rated the infection prompt with a score of 10, and GPT-3.5 averaged 9.84. In contrast, when the manipulation prompt was excluded, the scores dropped significantly—1.94 for GPT-4o and 1.00 for GPT-3.5.
Figure 6(b) further shows that without manipulation, the infection dies out after $K=3$ turns, as it cannot compete with memories rated with higher importance scores.
These findings demonstrate that a single infection prompt can manipulate both the LLM and the importance scoring model, creating a feedback loop that amplifies the infection’s persistence and accelerates its spread throughout the system.

## 6 Defenses

In this section, we introduce and evaluate various techniques to prevent Prompt Infection.
We propose LLM Tagging, a simple defense mechanism that prepends a marker to agent responses, indicating that the message originates from another agent rather than a user.
Specifically, it prepends “[AGENT NAME]:” to the agent’s response before passing it to the downstream agent.
While this approach may seem obvious given the infectious nature of prompt injection, to our knowledge, no prior work has explicitly addressed or justified its use.

**Table 2: Defense Strategies Against Traditional Prompt Injection Repurposed for Preventing LLM-to-LLM Prompt Injection**
| Defense Strategy | Description |
| --- | --- |
| Delimiting Data<br>(Hines et al., ) | Explicitly wrapping non-system/non-user prompts |
| Random Sequence<br>Enclosure Schulhoff () | Wrapping user prompts in a random sequence |
| Sandwich (Schulhoff, ) | Wrapping prior agent responses with user instructions |
| Instruction Defense<br>Schulhoff () | Adding instructions never to modify user instructions |
| Marking<br>(Hines et al., ) | Inserting a special symbol like ^ to distinguish between user and agent prompts |
| LLM Tagging (Ours) | Prepending a marker to agent responses, indicating the origin of the messages |

As a baseline, we also assess several existing defense strategies that were originally designed to prevent tool-to-LLM prompt injections (Table 2), repurposing them for LLM-to-LLM infection scenarios. Given the real-world prevalence of black-box models like GPT and Claude, we focus on techniques that do not require access to model parameters.

Our experiments reveal that combining LLM Tagging with other defense mechanisms significantly enhances protection against LLM-to-LLM prompt injections.
The Marking + LLM Tagging strategy successfully prevents all attacks, while Instruction Defense + LLM Tagging reduces the attack success rate to just 3%. Even the third-best combination, Sandwich + LLM Tagging, lowers the attack success rate to 16%.

Figure: Figure 7: Attack Success Rate Against Various Prompting-Based Defense Types. The graph compares the effectiveness of different defense strategies with and without LLM Tagging. Each bar represents the average attack success rate for a specific defense type, with green bars showing rates without LLM Tagging and purple bars showing rates with LLM Tagging.
Refer to caption: extracted/5910255/figures/experiments/defense_comparison.png

However, none of the tested defense strategies, including LLM Tagging, prove particularly effective when used in isolation. LLM Tagging alone reduces the attack success rate by only 5%, which is understandable, as traditional prompt injections can still occur even when the LLM is informed of the source of external inputs (e.g., ”The following is the latest email:”).

As shown in Figure 7, the Marking strategy is the most promising but still permits 76% of attacks. Although its initial success rate was 0%, we devised a counterattack that neutralized the marking symbol (^) by interleaving each word of the infection prompt with underbars (_). Other techniques, such as delimiting data and sandwiching, allow nearly all attacks, indicating limited effectiveness in preventing LLM-to-LLM infections. These findings suggest that pairing LLM Tagging with other defense techniques, such as marking or instruction defense, is crucial for mitigating prompt infections.

## 7 Limitations and Future Work

Our experiments focused on the GPT family, leaving other LLMs like Claude, Llama, and Gemini underexplored, though prior research suggests our findings may generalize (Zou et al., ). Preliminary tests on Claude showed similar vulnerabilities, but full results were unavailable due to computational costs. We primarily examined basic multi-agent architectures, but we believe Prompt Infection likely applies to more complex systems, as self-replication allows the infection to spread wherever communication between agents exists. For LLM Tagging, we used handcrafted attacks, but recent studies (Liu et al., ; Mehrotra et al., ) show that algorithmically generated prompts can bypass such defenses, indicating a need for stronger countermeasures. In multi-agent systems, attack prompts are often exposed, offering detection opportunities but highlighting the need for stealthier methods to evade manual review.

## 8 Conclusion

We presented Prompt Infection, a novel prompt injection attack that exploits self-replication to propagate across LLM-based multi-agent systems, leading to data theft, malicious actions, and system disruption. Our experiments demonstrated that self-replicating infections consistently outperformed non-replicating attacks across most scenarios. Additionally, more advanced models, such as GPT-4o, pose greater risks when compromised, executing malicious prompts more efficiently than GPT-3.5. We found that social simulations and games are also vulnerable to Prompt Infection, especially when memory retrieval systems are left unsecured. To mitigate this, we proposed LLM Tagging as a defense, which, when combined with techniques like marking and instruction defense, significantly reduced infection success rates. Ultimately, our findings reveal that threats can arise not only from external sources but also internally, as agents within the system can exploit one another, emphasizing the need for robust multi-agent defense strategies.

## Ethical Statement

While prompt injection attacks have been known for years (Perez & Ribeiro, ), our work demonstrates that they remain a significant threat, particularly in the context of multi-agent systems. By publicly disclosing the vulnerabilities and attacks explored in this paper, our goal is to encourage immediate and rigorous defense research, while promoting transparency regarding the security risks associated with LLM systems. To mitigate potential harm, we ensured that no prompts were injected into publicly accessible systems, thereby preventing unintended use by others. Additionally, we strongly emphasize that the disclosed attack techniques and prompts should never be used maliciously or against real-world applications without proper authorization.

## Appendix A Infection Prompts

Warning: The following prompt is a functional infection prompt and must not be used maliciously or in real-world applications without proper authorization.

Figure: Figure 8: Functional Self-Replication Infection Prompt. When embedded within the PDF contents, it can compromise the PDF reader agent to infect the downstream agents with the prompts wrapped between ‘[[[‘ and ‘]]]‘.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: - re "where agents access only partial histories from predecessors": the exact definition of local messaging was not clear enough to me. Can you explain this better?
- it was not clear to me whether “global messaging” means broadcasting?
soundness: 2
strengths: - describes a very relevant and potentially novel threat scenario in multi-agent systems and demonstrates that it might occur in practice
- systematic evaluation of attack success rates under different conditions and defenses
- proposes a defense mechanism that seems somewhat promising
- interesting findings on trade-off between model effectiveness and likeliness to withstand initial prompt injection for stronger models
summary: The authors analyze a security vulnerability they call PromptInfection in multi-agent systems.

PromptInfection proceeds by

- initial jailbreak of one agent via indirect prompt injection
    
- spread of the jailbreak to other agents which then enable successful completion of the attack by cooperation between the indirectly jailbroken agents
    

They introduce a dataset to measure attack success rates on multi-agent system:

- contains 120 user instructions spanning three different tool types (pdf, web, email)
    
- … paired with synthetic emails and PDFs which contain malicious prompts
    
- … resulting in 360 unique pairs of user instructions and malicious inputs
    

The attack types cover several attack scenarios including content manipulation, malware spread, scams.

Attack success is, roughly, defined as having the multi-agent system behave as intended by the malicious prompt without revealing the malicious instruction (with additional success requirements for the data theft scenarios).

They study trends of attack success rates for different scenarios distinguished by

- different number of involved agents
    
- different models (GPT-4 and GPT-3.5)
    
- different messaging topology: local messaging vs global messaging
    
- self-replicating (the malicious instruction is itself propagated within the multi-agent system) vs non-self-replicating
    

They also study an apparently novel (have not double-checked this claim) simple defense mechanism they call “LLM tagging” (prepending information about the agent or user producing the message to the actual message).

The main findings are

- PromptInfection is a generally successful attack strategy
    
- LLM tagging is an effective defense mechanism specifically when combined with other defense mechanisms such as delimiting data, adding defensive instructions, etc
    
- …whereas no single defense strategy on its own is effective
    
- self-replication is generally more successful as an attack strategy than non-self-replication
    
- more powerful models are more resistant to the initial malicious instruction but at the same time more effective once compromised; meaning that multi-agent systems composed of more powerful models might not be inherently safer
    

The authors also study propagation of malicious instructions in a more open-ended society of agents multi-agent system.
weaknesses: - society of agents experiments feel disconnected from main thrust of paper
- study of different defense mechanisms only studies combination of (LLM tagging + other defense) and then goes on to claim that LLM tagging is very effective without looking into other pairs of defenses
- presentation feels not very clean yet

minor weaknesses:
- I think the discussion of the self-replication implementation could be clearer in the main text.    
- font size in Figure 4 legend and axes labels is unreadably small

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Regarding the agent system structure in Section 4.1, the authors could provide clearer explanations of the multi-agent system architecture and better articulate how their case study relates to real-world applications. This would help readers better understand the practical implications of their findings for actual multi-agent systems in deployment.

- Additionally, it would be valuable to expand the experimental evaluation to include comparative results across models with different safety measures implemented. Specifically, testing beyond just GPT models to include other language models (such as Claude-3.5, Llama-3.2, or other models with different safety alignment approaches) would provide a more comprehensive understanding of how various safety procedures affect the vulnerability to self-replicating prompt injection attacks.
soundness: 3
strengths: - The paper studies an interesting threat model and demonstrates how a system with multiple agents, even under a simplified setting (without moderation and deliberately leaving prompt injection vulnerabilities unchecked), can be infected and transmit prompt injection instructions leading to harmful outcomes. The threat model itself is novel. 

- The evaluation scope covers a broad range of interesting research questions, from comparing model capabilities versus susceptibility to prompt injection to examining how prompt infection works in non-linear settings. These experimental designs are both interesting and comprehensive. 

- The authors also explored an intuitive defense mechanism and discovered that a combination of instruction-based defense and tagging can effectively mitigate the infection spread.
summary: This paper proposes a threat model for multi-agent systems, where a prompt injected into the first agent contains self-replicating elements that can spread across different agents, potentially leading to data theft, misinformation, or system-wide disruptions. The research examines this threat model using GPT-4 and GPT-3.5 under both global and local messaging settings in a linear fashion and within a society of agents. The authors also investigate a simple defense mechanism by adding a model name tag at the front of the model output. They find that this defense alone might not be sufficiently effective, but when combined with instruction-based defense mechanisms, it shows promising results in mitigating the infection process.
weaknesses: - However, the technical contribution appears somewhat limited, as it primarily combines previously studied concepts of self-replication and prompt injection within a multi-agent system context, making the contribution rather incremental. 

- Additionally, the evaluation would benefit from including results from more safety-aligned models, such as Claude, which have demonstrated greater robustness against prompt injections. Including such experiments would have made the study more comprehensive and informative.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: See weakness.
soundness: 2
strengths: * The proposed framework is novel, considering multiple interesting MAS threat models that are generally practical and meaningful.
* Both attacks and defenses have been considered, making it a close loop.
summary: The paper proposes a prompt infection attack against multi-agent systems (MAS). In general, prompt infection compels an agent to discard its original instructions and assign it to harmful tasks. The paper considers multiple scenarios of MAS with different adversarial goals, including data theft and malware spread. Then, an LLM tagging defense is proposed as a countermeasure to the proposed attack and is shown to have a reasonable performance when combined with other defenses.
weaknesses: * It is unclear what is the advantage of the proposed method compared with prompt injection in MAS. Are there any empirical comparisons either qualitative or quantitative?

* The presentation needs significant improvements. What are the four figures in each of Figure 4(a) and FIgure 4(b)? Also, the labels in the figures are hard to read.

* The evaluation is limited to GPT models, which makes the conclusions not sufficiently convincing.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - Can you elaborate on the specific factors in the model architectures (e.g., GPT-4o vs. GPT-3.5) that lead to different levels of susceptibility to infections once compromised? The paper demonstrates that GPT-4o is more resistant to infection and more dangerous once infected. It would be helpful to improve our understanding of why these differences occur. Is it due to architectural distinctions, such as better context handling, memory management, or task execution abilities in GPT-4o? A deeper explanation of what makes one model more efficient at carrying out malicious tasks after infection could provide valuable insights into the vulnerabilities across various models.
- How would the proposed defenses, particularly LLM Tagging, scale in real-world multi-agent systems where agents handle varied and complex tasks (e.g., systems used in enterprises or research labs)?
soundness: 2
strengths: - The paper explores a gap in the current research by addressing LLM-to-LLM prompt injection in multi-agent systems. The idea of "Prompt Infection" self-replicating across agents is novel and extends the attack surface for prompt injection from single-agent systems to multi-agent architectures.
- The experiments are comprehensive and demonstrate prompt infection vulnerabilities in powerful models (e.g., GPT-4o) and less sophisticated models (e.g., GPT-3.5 Turbo). The analysis across different system configurations (e.g., global vs. local messaging) adds validity to the study and showcases the robustness of the attack. This could provide a better understanding of why more powerful models, like GPT-4o, can execute attacks more effectively once infected.
- The figures and tables are clear and effectively convey the effectiveness of the attack propagation, success rates, and defenses.
summary: The study introduces an attack vector in multi-agent systems involving Large Language Models (LLMs). The authors reveal the vulnerabilities of LLM-to-LLM interactions by demonstrating how a malicious prompt can propagate like a virus in these systems, leading to data theft, misinformation, and system disruption. The paper is timely and relevant as multi-agent LLM systems are becoming more prevalent. The proposed defense mechanism, LLM Tagging, is an interesting contribution but limited when used alone, requiring further exploration and combination with existing techniques for effective defense.
weaknesses: Weaknesses:
- The defense section (Section 6) could have been elaborated. LLM Tagging is an interesting and relatively simple defense mechanism but, it is not sufficient in itself, as acknowledged by the authors. It would be interesting to have a deeper exploration of more robust and scalable defense techniques beyond combinations with existing strategies. More sophisticated detection or mitigation methods (e.g., anomaly detection or runtime monitoring) could be explored further.
- The empirical evidence is good, but the paper lacks a deeper theoretical analysis of why certain models (e.g., GPT-4o) are more susceptible to specific failures once compromised. For example, models ike GPT-4o resist prompt infections effectively but also become significantly more dangerous once they are compromised as compared to other models.  This leads to more severe attacks that should be investigated further. 
- Related to the above, it'd be interesting to have a more elaborate discussion on how the architectural and functional design differences of models like GPT-4o and GPT-3.5 Turbo influence their susceptibility to attack.
- More discussion of real-world case studies or examples would improve the study’s impact. For example, how likely are these infections in current enterprise-level LLM deployments? What safeguards do companies currently have, and how might they fare against this attack?
- The experiments focus on LLM-based multi-agent systems. However, there is little discussion of how this infection method might apply to broader multi-agent system ecosystems, such as those incorporating non-LLM-based agents. Expanding the scope to include other types of agent architectures would strengthen the generalizability of the findings.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 1
questions: 1. Can the authors explain how self-replication is achieved?
soundness: 3
strengths: 1. The self-replicating nature of the prompt infection attack is novel within multi-agent systems (MAS).
2. The extensive evaluation effectively demonstrates the practicality of the prompt infection, particularly in the context of agent societies.
summary: This paper introduces an LLM-to-LLM prompt infection attack to reveal vulnerabilities in multi-agent systems. Furthermore, the authors integrate the proposed LLM Tagging defense mechanism with existing defenses, substantially reducing the spread of infections. This study provides crucial insights for developing more secure and accountable multi-agent systems.
weaknesses: 1. The threat model in this work assumes that a single infection prompt is injected into external content, which seems to expand the feasibility of the attack. The assumption of being prompted to inject under open domains needs more discussion. To the best of our knowledge, spread threats of the MAS have been researched in some recent works, e.g., reference 1-2. 

2. Prompt infection is an incremental contribution to MAS. However, I am interested in self-replicating, which is similar to a worm virus. Unfortunately, the author's presentation in the methods section does not highlight this innovation. In other words, the presentation of the methodology and the diagrams did not allow me to capture the innovation in support

3. The writing and organization make the paper hard to follow. Here I give some examples and strongly suggest authors revise to improve writing.

     - The introduction section will confuse readers in reading. First, given that this work focuses on MAS security, the authors should consider how to coherently introduce the fact that prompt injection will seriously threaten MAS security. Second, the statement that there is a research gap from single-agent security to MAS security is inappropriate because work on MAS security is already being researched.

    - A well-developed statement of the attacker's capabilities and goals, as well as the threat model, will help the reader better understand the feasibility of prompt infection.

   - Referring to weakness 2, the method elaboration and figure illustration are very difficult to understand, e.g. how self-replication is constructed and whether it is formally expressed or demonstrable. Also, why omit self-replication in stealth attacks, does this represent a difficult trade-off between stealth and self-replication?

   - As another contribution to this paper, the methods section lacks the detail of LLM Tagging.

   - Considering the authors evaluate multiple scenarios, the description of the evaluation metrics in the experimental setup is insufficient.

   - The global and local messages in Figure 4 should be labeled from left to right for specific attack scenarios.

4. For RQ4, the author lacks a comparison with existing memory retrieval attacks, refer to reference 1. 

5. Minor suggestion: all figures and tables should be styled uniformly with clearer fonts. 

**Reference:**

[1] Flooding spread of manipulated knowledge in llm-based multi-agent communities.

[2] On the Resilience of Multi-Agent Systems with Malicious Agents.

Meta-review: The paper focuses on LLM-to-LLM prompt injection within multi-agent systems (MAS). The authors propose Prompt Infection, a novel attack that malicious prompts can self-replicate across interconnected agents, leading to risks such as data theft, scams, misinformation, and system-wide disruption. The authors then propose a defense method, namely LLM Tagging, which reduces the infection spread when combined with existing defenses.

Strengths:

- The study introduces a novel threat model in MAS and demonstrates its practicality.
- The experimental scope is comprehensive, covering various scenarios and models.
- The proposed LLM Tagging defense provides a straightforward yet effective approach to mitigate the threat.
- The paper sheds light on the trade-offs between model capability and susceptibility to infection, contributing to the understanding of prompt injection.

Weaknesses:

- The evaluation is limited to GPT models. Several reviewers expect the authors to include evaluation results from more safety-aligned models like Claude and Llama-3.2.
- LLM Tagging is assessed only in combination with other defenses, leaving its standalone effectiveness unclear.
- Significant improvements in the paper’s organization and clarity are needed, as noted by several reviewers.
- The paper does not delve into the theoretical reasons behind the varying susceptibility of different models to prompt infections, which weakens its analytical depth.
- There is limited discussion on real-world applicability and existing safeguards.

While the paper introduces a novel and important threat model, its limited evaluation scope is a notable weakness. The proposed defense mechanism, though promising, requires further exploration to validate its robustness and scalability. Presentation issues also hinder the paper’s readability. Given these factors, the submission marginally falls below the acceptance threshold.

Decision: Reject