GuardAgent: Safeguard LLM Agent by a Guard Agent via Knowledge-Enabled Reasoning

Authors: Zhen Xiang, Linzhi Zheng, Yanjie Li, Junyuan Hong, Qinbin Li, Han Xie, Jiawei Zhang, Zidi Xiong, Chulin Xie, Carl Yang, Dawn Song, Bo Li
Primary area: foundation or frontier models, including LLMs
Keywords: LLM agent, guardrail, safety, LLM

Abstract:
The rapid advancement of large language models (LLMs) has catalyzed the deployment of LLM-powered agents across numerous applications, raising new concerns regarding their safety and trustworthiness. In addition, existing methods for enhancing the safety of LLMs are not directly transferable to LLM-powered agents due to their diverse objectives and output modalities. In this paper, we propose
GuardAgent, the first LLM agent as a guardrail to protect other LLM agents.
Specifically, GuardAgent oversees a target LLM agent by checking whether its
inputs/outputs satisfy a set of given guard requests, e.g., safety rules or privacy
policies defined by the users. The pipeline of GuardAgent consists of two steps: 1) create a task plan by analyzing the provided guard requests, and 2) generate
guardrail code based on the task plan and execute the code by calling APIs or
using external engines. In both steps, an LLM is utilized as the core reasoning
component, supplemented by in-context demonstrations retrieved from a memory
module storing information from previous sessions. Such knowledge-enabled reasoning of GuardAgent allows it to understand various textual guard requests and
accurately “translate” them into executable code that provides reliable guardrails.
Furthermore, GuardAgent is equipped with an extendable toolbox containing
relevant APIs and functions, and requires no additional LLM training, underscoring
its flexibility and low operational overhead. In addition to GuardAgent, we
propose two novel benchmarks: an EICU-AC benchmark for assessing privacy-
related access control for healthcare agents and a Mind2Web-SC benchmark for
assessing safety regulations for web agents. When using Llama3-70B/Llama3.1-
70B/GPT-4 as the core LLM, GuardAgent achieves 98.4%/98.4%/98.7% and
83.5%/84.5%/90.0% guarding accuracy on these two benchmarks in moderating
invalid inputs and outputs of two types of agents, respectively. We also show the
ability of GuardAgent to define necessary functions that are absent from the
toolbox, which further highlights the flexibility of GuardAgent in adaption to
new LLM agents and guard requirements.

Decision: reject
Reviewer scores: 6, 5, 5, 8

Reviews and discussion:
<thread>
creation date:
1730227317933

thread_type:
official_review

<thread_object>
creator: reviewer_codc

<content>
summary:
This paper focuses on guardrails for LLM agents, with the authors proposing GuardAgent, a LLM agent designed to safeguard other LLM agents based on specified guard requests from users. The paper has the following contributions: (1) development of an LLM agent framework to guard other target LLM agents based on user requirements and using a memory module that stores previous use cases; (2) creation of two benchmarks for healthcare and web agents to evaluate access control policies; (3) evaluation and ablation study evaluating GuardAgent vs. baseline models.

strengths:
This paper presents an innovative framework for a LLM agent to safeguard other LLM agents. The design of the framework is noninvasive, generates guardrails using python code execution (not natural language) and does not require LLM training or retraining, which are all strengths. Using python code execution as opposed to some knowledge-specific languages as done in some other works (e.g., Kang, Li R^2-Guard) is a bonus, supporting adoption by a wider set of programmers/engineers. 

In addition, the integration of the memory module is a nice feature which allows the agent to document new cases it runs into and learn from previous examples which can help reduce the burden of a user needing to specify an exhaustive list of requirements, and may allow for adaption to new (unseen) scenarios. This is an innovation compared to previous work (Rebedea et al. NeMo Guardrails; Inan et al, Llama Guard; Ghosh et al. Aegis) which either require users to specify and provision every safety/privacy property or need some type of training guideline, such as classification labels. The paper evaluates the framework based on the curated benchmarks and compared to baselines for different LLM model types. The experiments make sense as they test precision, recall and two measures of accuracy for the labels about whether a policy is violated or not. The eval includes an ablation study which shows performance improvements for the inclusion of the memory module and toolbox (supporting framework architecture decisions), and provide evidence that the framework performs well with performance improvements compared to the baselines in almost all criteria.

Finally, this paper is presented very nicely- the structure, organization and language are very clear and the figures are unambiguous and helpful.

weaknesses:
The LLM guardrail space is quite saturated with recent work, and the related work section is quite concise. It might be worth expanding this section to better motivate the need for this work. For example, the authors mention that model- and agent- guarding cannot be directly used to safeguard  LLM agents with diverse output modalities. Can the authors provide some contextual examples or explain why?

The access control policies used for the healthcare and web agent scenarios and evaluation (though relevant for these application areas) are pretty simplistic (see question 1)). In many real world use cases, more complicated policies may be required to accurately provision safety/privacy constraints. 

The GuardAgent only returns binary (policy violated or not) responses, which may not be sufficient to represent all cases of safety and privacy policies. Many policies are not so strict, and may need to return risk-based or threshold based responses (see question 2).

A pro of the framework is the automatic code generation through use of the framework's toolbox. I am wondering about the reliability and generalizability of such a process as inexecutable generated code would render the framework useless (see question 3). This is not directly evaluated in the experimental section (though the authors mention code is almost always executable and does not often use the debugging stopgap in Section 4.3). 

It would be great if the authors intended to release their framework and benchmarks (e.g., so that people could contribute to the toolbox functionalities or specifications).

questions:
(1) Can the authors provide examples or comment on the ability of the framework to adapt to more complicated policies than the ones evaluated in the experimental section (such as more fine-grained access controls for different users)?

(2) Can the authors comment on the ability of GuardAgent to handle nonbinary requirements (e.g., safety requirements that work on a threshold, risk-based requirements, etc.)? 

(3) Can the authors comment on or provide evidence that the framework's code generation process is reliable and generalizable (i.e., generates low rates of inexecutable code even for new safety or privacy policies or application domains)? Under what scenarios or tasks might this change; do new policies that have limited to no occurrence in the memory module result in more inexecutables?

(4) Do the authors have an idea about why the recall LPR does better for GPT-4 baseline compared to GuardAgent (Table 2)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730587010487

thread_type:
official_review

<thread_object>
creator: reviewer_1tyk

<content>
summary:
This paper presents GuardAgent, a framework designed to enhance the safety and robustness of LLMs against adversarial inputs and potential misuse. GuardAgent leverages a multi-agent architecture, incorporating safety-checking agents that detect unsafe responses, and employs prompt engineering to guide the LLM toward safer outputs. The framework integrates several defensive techniques, including uncertainty estimation, self-refinement prompts, and cross-agent validation, to achieve a high level of safety without compromising the model’s performance. In addition to GuardAgent, this paper also proposes two novel benchmarks: an EICU-AC benchmark for assessing privacy-related access control for healthcare agents and a Mind2Web-SC benchmark for assessing safety regulations for web agents.

strengths:
1. GuardAgent is the first framework focused on providing guardrails to LLM agents, addressing a critical gap in AI agent safety and privacy.

2. The system's non-invasive approach and extendable toolbox make it adaptable to diverse LLM agents and new guard requests.

3. GuardAgent demonstrates superior accuracy and reliability compared to baseline models, particularly when using code-based guardrails.

weaknesses:
1. **Dataset Scope**: The paper evaluates GuardAgent on two specific datasets, EICU-AC for privacy-related access control in healthcare and Mind2Web-SC for safety compliance in web agents. While these datasets represent different types of guard requests, the limited scope raises questions about GuardAgent's generalizability. As a defense mechanism, how effectively can GuardAgent adapt to other domains beyond healthcare and web safety, where guardrails may vary significantly?

2. **Performance Variability**: The paper evaluates GuardAgent using different core LLMs, such as Llama3-70B and GPT-4, but it does not fully clarify how this choice impacts GuardAgent’s overall effectiveness. Does GuardAgent’s performance, including its accuracy in identifying violations, speed in generating guardrail code, and flexibility in adapting to complex guard requests, vary significantly depending on the model used?

3. **Memory Dependency**: The ablation study shows that GuardAgent’s performance improves with more in-context demonstrations. How does the quality of these demonstrations, such as their relevance or diversity, impact accuracy? Is there an optimal number of demonstrations that balances accuracy and efficiency, and does this vary by task or application?

4. **Computational Efficiency**: What are the computational costs associated with GuardAgent’s code generation and execution process? 

5. **Error Handling**: GuardAgent includes a debugging mechanism that uses an LLM to analyze and address errors during code execution. Could you clarify how robust this mechanism is? Specifically, how well does it handle different error types (e.g., syntax errors, logical errors, unforeseen inputs), and are there limits to the errors it can reliably resolve? Additionally, has it been tested in scenarios with complex or ambiguous errors, and what are its typical failure modes, if any?

questions:
See the Weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730699766332

thread_type:
official_review

<thread_object>
creator: reviewer_4xot

<content>
summary:
This paper presents GuardAgent, a protective guardrail framework that functions as a third-party safeguard for other LLM agents. GuardAgent initially utilizes an LLM to develop an action plan derived from guard requests as well as the inputs and outputs of the target agent. The LLM then transforms this plan into guardrail code, which is subsequently executed by an external engine. The authors also introduce two benchmarks specifically aimed at evaluating LLM safety: EICU-AC, designed to test access control for LLM agents in healthcare, and Mind2Web-SC, a dataset intended to assess safety mechanisms for web agents powered by LLMs. Experimental validation on these datasets shows that GuardAgent performs better than the "model-guard-agent" baseline, which uses hard-coded task instructions.

strengths:
**S1**: A valuable contribution of this paper is providing the LLM community with the probably first safety benchmarks that explicitly model user profiles. In these two benchmarks, the actions that the LLM agent needs to perform are related to the user's identity and profile, requiring the agent to assess whether the user's actions may pose potential risks based on their identity. These datasets present a greater challenge for LLM agents, as they need to incorporate the user profile and assigned permissions into the context to complete tasks while minimizing risks. Such benchmarks closely align with real-world needs, specifically in providing different services based on varying user permissions. This approach offers the LLM community a new perspective and dimension, while also presenting new challenges.

**S2**: In tasks aimed at enhancing the alignment or safety capabilities of LLMs, safety and helpfulness are often in conflict, requiring a trade-off. Specifically, as the safety capabilities of LLMs increase, the likelihood of them generating refusal responses also increases, which in turn reduces the likelihood of providing helpful information to the user—a phenomenon known as the "safety tax." However, in this paper, experiments demonstrate that GuardAgent does not affect the task performance of the target agent while safeguarding against potential risks. This design represents a well-executed approach.

weaknesses:
**W1**: The work in this paper first designs two benchmarks, i.e., EICU-AC and Mind2Web-SC, and then develops the method based on these two tasks. However, the motivation for proposing these benchmarks has not been well justified. Why are database retrieval (EICU-AC) and web service calls (Mind2Web-SC) the two representative tasks for testing the safety of LLM agents? What makes these two tasks sufficiently representative to measure the safety capabilities of LLM agents in mitigating risks? Are there any similar tasks in previous research that have been used to validate the safety of LLM agents? Why introduce these new tasks instead of using existing benchmarks? In summary, there needs to be an explanation and justification for the motivation behind proposing these new benchmarks.

**W2**: At the beginning of this paper, one contribution is introduced as ``generate guardrail code based on the task plan." However, after reading the benchmark design section, we find that the so-called guardrail code generation is actually a task-oriented function, not a task-agnostic, universally applicable design. Since the EICU-AC task itself requires generating structured query code for database retrieval, GuardAgent needs to have code generation capabilities. This doesn’t mean that a unique code design was developed to enhance the safety of LLM agents. Here, if the downstream task doesn’t require code generation, such as in a complex Q&A task where responses are given in natural language, would code generation then be a redundant design for GuardAgent? 

**W3**: Additionally, the authors state that GuardAgent leverages the LLM's reasoning capabilities to "accurately ‘translate’ textual guard requests into executable code." However, this capability is task-dependent. When the task does not require code generation, wouldn’t this reasoning ability be unnecessary? Therefore, the contribution proposed in this paper—particularly the guardrail code generation—is determined by the characteristics of the downstream tasks. It is not inherently a design that can be applied to various tasks to enhance LLM agent safety, which weakens this work's generalizability and impact.

questions:
**Q1**: It is better to briefly introduce the definition of ‘model-guard-agent’ baseline when it first appears in this paper. Otherwise, it will confuse the readers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730719317283

thread_type:
official_review

<thread_object>
creator: reviewer_2hqu

<content>
summary:
This paper proposes GuardAgent, the first LLM agent designed to safeguard other LLM agents. GuardAgent utilizes the reasoning capabilities of LLMs to generate a task plan and translate it into guardrail code. It stands out for its flexibility in handling diverse guardrail requests by retrieving relevant demonstrations from a memory module, its reliability through code-based guardrails, and its low computational overhead, requiring no additional LLM training.

strengths:
• The paper is well-structured, with a clear logical flow that facilitates understanding. The experimental design is concise, and the results are presented in an easily interpretable manner, enhancing the clarity of the study.

• The paper effectively underscores the necessity of the proposed "agent guarding agent" approach, particularly highlighting GuardAgent's importance in accommodating dynamic and complex guardrail requests. This emphasis supports the relevance and timeliness of the guardrail framework in addressing complex safety and privacy challenges.

weaknesses:
• Lack of experimental comparison with existing works on Guardrail methods. While the related work section discusses existing many guardrail approaches, the study conducts only a brief comparison with “model guarding agent” approach. Expanding the scope of comparison with a broader range of guardrail techniques would strengthen the evaluation of GuardAgent’s effectiveness.

• In the ablation studies, the authors mention “the trend of code-based guardrails” as a rationale for the code-generation design of GuardAgent, but this observation has only been briefly mentioned. This aspect appears intriguing, and further experimental analysis and a more detailed discussion would enhance the understanding of this design choice.

• As the first work to explore “agent guarding agents”, this paper is positioned to serve as a key reference for future research in this domain. However, to support subsequent studies, it would benefit from an analysis of the GuardAgent limitations or potential threats to validity associated with current version of paper, as well as a discussion of possible future directions on “agent guarding agent”  to further advance the development of robust guardrail frameworks.

• Some typos are found.
- Section 4.1: In the second step 4.3),-> In the second step (Sec. 4.3)
- Why all metrics in all Tables has upwards arrow?
- Section 5.3: Tab. 2-> Table 2, Fig. 3-> Figure 3
- Please give a formal(short) caption for all Figures and Tables

questions:
1. Could you clarify which specific work is being referred to as "agent guarding models" approach？
2. Additionally, it appears that related work on "model guarding agents" approaches has not been cited, right?
3. Could you discuss more about current limitations/challenges and future direction of “agent guarding agent” approach?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732296596088

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely thank the reviewers for their constructive feedback and valuable suggestions. We are pleased that the reviewers recognize the importance of our work and its success in highlighting the necessity of "agent guarding agent" approaches. We also greatly appreciate the acknowledgment of our contributions, including the proposal of two novel benchmarks, the thorough evaluation of our proposed GuardAgent framework on these benchmarks, and the quality of our presentation.  Below, we outline the key changes we have made in response to the reviewers’ comments. The paper has been updated and the revision is highlighted in blue.

We have added the following experiment results:
* Added evaluations and results (with analysis and case studies) on two complex QA datasets MMLU and CSQA to show the advantages of code-based guardrail, the generalization of GuardAgent framework, and GuardAgent’s capabilities to handle non-binary safety policies. (Reviewers 4xoT, 1TYK, and Codc)
* Added evaluations to show that guardrails for models such as LlamaGuard3 cannot safeguard LLM agents. (Reviewer 2Hqu)
* Added execution time of GuardAgent for more core LLMs beyond GPT-4. (Reviewer 1YTK)
* Added evaluation results to support GuardAgent’s memory retrieval policy based on max-similarity. (Reviewer 1YTK)
* Added evaluation results with analysis to show the reliability of GuardAgent’s code generation and its generalization to new policies when there are tools or memory. (Reviewer Codc)
* Added experiments and evaluations to analyze the effectiveness of the debugging process of GuardAgent. (Reviewer 1YTK)
* Added more evaluation results and analysis to explain the “trend of code-based guardrails”. (Reviewer 2Hqu)

We have also added more discussion and analysis:
* Expanded the related work section to: (Reviewers 2Hqu and Codc)
  - Clarify we are the first to investigate both the “model guarding agent” and “agent guarding agent” approaches.
  - Explain why guardrails for LLMs cannot be directly used to safeguard LLM agents.
  - Describe the “agent guarding model” approach.
* Added a section for limitations and future research and explained how our framework can be adapted to more complicated safety policies. (Reviewers 2Hqu nad Codc)
* Added discussion about the motivation of the two proposed benchmarks. (Reviewer 4xoT)
* Added discussion about the impact of the core LLM of GuardAgent on its efficacy and flexibility. (Reviewer 1TYK)
* Added discussion about how the relevance of retrieved demonstrations impacts the efficacy and efficiency of GuardAgent. (Reviewer 1TYK)

We hope all of the reviewers’ concerns have been well-addressed in our responses. We are more than willing to address additional questions and conduct further experiments should the reviewers deem it necessary.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents GuardAgent - an "agent guarding agents" approach, as opposed to "models guarding agents". 
The code-based guardrail is designed to be more robust to accomodate guardrail requests, in comparison to natural-language based guardrails. GuardAgent responds to an explicit list of guardrail requests specified by the users. 
The method is evaluated on two tasks and datasets: database retrieval (EICU-AC) and web service calls (Mind2Web-SC) as the two representative tasks for testing the safety of the specific agents: medical agents and web agents.
The guardrail code generation is thus task-specific to the datasets and chosen benchmark tasks.
The authors created a set of safety rules for each dataset a original datasets did not include any rules - to test the QA tasks.
Overall the paper is well written and the experiments part are clear.

However, it is unclear still why the authors did not use well established QA tasks used for LLM safety, since they also introduced safety rules for the original datasets. This is a question by reviewer 4xoT and the responses by authors seem to have not fully addressed the questions on the specificity of the tasks, and in other scenarios when safety guardrails are not explicitly stated, or in other Q&A datasets/ benchmark tasks. The fact that the paper only is evaluated on two very specific datasets and agentic tasks is the major limiting factor. 

The authors were very responsive during the rebuttal period which is highly commendable.
Although two of the reviewers increased their score, this paper still falls exactly on the borderline mark. Although there is a champion reviewer, two of the reviewers have score 5, and with one of the active reviewers and expert is adamant that the paper has not quite passed the borderline. 

Therefore, I am recommending rejection for this paper and suggest the authors to polish this paper further for future submission.