GuardAgent: Safeguard LLM Agent by a Guard Agent via Knowledge-Enabled Reasoning

## Abstract

Abstract The rapid advancement of large language models (LLMs) has catalyzed the deployment of LLM-powered agents across numerous applications, raising new concerns regarding their safety and trustworthiness.
In addition, existing methods for enhancing the safety of LLMs are not directly transferable to LLM-powered agents due to their diverse objectives and output modalities.
In this paper, we propose GuardAgent , the first LLM agent as a guardrail to other LLM agents.
Specifically, GuardAgent oversees a target LLM agent by checking whether its inputs/outputs satisfy a set of given guard requests (e.g., safety rules or privacy policies) defined by the users. GuardAgent comprises two steps: 1) creating a task plan by analyzing the provided guard requests, and 2) generating guardrail code based on the task plan and executing the code by calling APIs or using external engines.
In both steps, an LLM is utilized as the core reasoning component, supplemented by in-context demonstrations retrieved from a memory module.
Such knowledge-enabled reasoning allows GuardAgent to understand various textual guard requests and accurately “translate” them into executable code that provides reliable guardrails.
Furthermore, GuardAgent is equipped with an extendable toolbox containing functions and APIs and requires no additional LLM training, which underscores its generalization capabilities and low operational overhead.
In addition to GuardAgent , we propose two novel benchmarks: an EICU-AC benchmark for assessing privacy-related access control for healthcare agents and a Mind2Web-SC benchmark for safety evaluation for web agents.
We show the effectiveness of GuardAgent on these two benchmarks with 98.7% and 90.0% guarding accuracy in moderating invalid inputs and outputs for the two types of agents, respectively.
We also show that GuardAgent is able to define novel functions in adaption to emergent LLM agents and guard requests, which underscores its strong generalization capabilities.

## 1 Introduction

AI agents empowered by large language models (LLMs) have showcased remarkable performance across diverse application domains, including finance [], healthcare [, , , , ], daily work [, , , ], and autonomous driving [, , ].
For each user query, these agents typically employ an LLM for task planning, leveraging the reasoning capability of the LLM with the optional support of long-term memory from previous use cases [].
The proposed plan is then executed by calling external tools (e.g., through APIs) with potential interaction with the environment [].

Unfortunately, the current development of LLM agents primarily focuses on their effectiveness in solving complex tasks while significantly overlooking their potential for misuse, which can lead to harmful consequences.
For example, if misused by unauthorized personnel, a healthcare LLM agent could easily expose confidential patient information [].
Indeed, some LLM agents, particularly those used in high-stakes applications like autonomous driving, are equipped with safety controls to prevent the execution of undesired dangerous actions [, ].
However, these task-specific guardrails are hardwired into the LLM agent and, therefore, cannot be generalized to other agents (e.g., for healthcare) with different guard requests (e.g., for privacy instead of safety).

On the other hand, guardrails for LLMs provide input and output moderation to detect and mitigate a wide range of potential harms [, , , , ].
This is typically achieved by building the guardrail upon another pre-trained LLM to contextually understand the input and output of the target LLM.
More importantly, the ‘non-invasiveness’ of guardrails, achieved through their parallel deployment alongside the target LLM, allows for their application to new models and harmfulness taxonomies with only minor modifications.
However, LLM agents are significantly different from LLMs, as they involve a much broader range of output modalities and highly specific guard requests.
For instance, a web agent empowered by LLM might generate actions like clicking a designated button on a webpage [].
The guard requests here could involve safety rules that prohibit certain users (e.g., those under a certain age) from purchasing specific items (e.g., alcoholic beverages).
Clearly, existing guardrails designed solely to moderate the textual inputs and outputs of LLMs cannot address such intricate guard requests.

In this paper, we present the first study on guardrails for LLM agents.
We propose GuardAgent, the first generalizable framework that uses an LLM agent to safeguard other LLM agents (referred to as ‘target agents’ henceforth) by adhering to diverse real-world guard requests from users, such as safety rules or privacy policies.
The deployment of GuardAgent requires the prescription of a set of textural guard requests informed by a specification of the target agent (e.g., the format of agent output and logs).
During the inference, user inputs to the target agent, along with associated outputs and logs, will be provided to GuardAgent for examination to determine whether the guard requests are satisfied or not.
Specifically, GuardAgent first uses an LLM to generate an action plan based on the guard requests and the inputs and outputs of the target agent.
Subsequently, the LLM transforms the action plan into a guardrail code, which is then executed by calling an external engine.
For both the action plan and the guardrail code generation, the LLM is provided with related demonstrations retrieved from a memory module, which archives inputs and outputs from prior use cases.
Such knowledge-enabled reasoning is the foundation for GuardAgent to understand diverse guard requests for different types of LLM agents.
The design of our GuardAgent offers three key advantages.
Firstly, GuardAgent can be easily generalized to safeguard new target agents by simply uploading new functions to its toolbox.
Secondly, GuardAgent provides guardrails by code generation and execution, which is more reliable than guardrails solely based on natural language.
Thirdly, GuardAgent employs LLMs by in-context learning, enabling direct utilization of off-the-shelf LLMs without the need for additional training.

Before introducing GuardAgent in Sec. 4, we investigate diverse guard requests for different types of LLM agents and propose two novel benchmarks in Sec. 3.
The first benchmark, EICU-AC, is designed to assess the effectiveness of access control for LLM agents for healthcare.
The second benchmark, Mind2Web-SC, evaluates safety control for LLM-powered web agents.
These two benchmarks are used to evaluate our GuardAgent in our experiments in Sec. 5.
Note that the two types of guard requests considered here – access control and safety control – are closely related to privacy and safety, respectively, which are critical perspectives of AI trustworthiness [].
Our technical contributions are summarized as follows:

- •
We propose GuardAgent, the first LLM agent framework providing guardrails to other LLM agents via knowledge-enabled reasoning in order to address diverse user guard requests.
- •
We propose a novel design for GuardAgent, which comprises knowledge-enabled task planning using in-context demonstrations, followed by guardrail code generation involving an extendable array of functions.
Such design endows GuardAgent with strong generalization capabilities, reliable guardrail generation, and no need for additional training.
- •
We create two benchmarks, EICU-AC and Mind2Web-SC, for evaluating privacy-related access control for healthcare agents and safety control for web agents, respectively.
- •
We show that GuardAgent effectively provides guardrails to 1) an EHRAgent for healthcare with a 98.7% guarding accuracy in access control and 2) a SeeAct web agent with a 90.0% guarding accuracy in safety control.
We also demonstrate the capabilities of GuardAgent in defining new functions during guardrail code generation and execution.

## 2 Related Work

LLM agents refer to AI agents that use LLMs as their central engine for task understanding and planning and then execute the plan by interacting with the environment (e.g., by calling third-party APIs) [].
Such fundamental difference from LLMs (with purely textual outputs) enables LLM agents to be deployed in diverse applications, including finance [], healthcare [, , , , ], daily work [, , , ], and autonomous driving [, , ].
LLM agents are also commonly equipped with a retrievable memory module, allowing them to perform knowledge-enabled reasoning to handle different tasks within its application domain [].
Our GuardAgent is a typical LLM agent, but with different objectives from existing agents, as it is the first one to safeguard other LLM agents.

LLM-based guardrails belong to a family of moderation approaches for harmfulness mitigation [, ].
Traditional guardrails were operated as classifiers trained on categorically labeled content [, ], while recently, guardrails based on LLMs with broader contextual understanding have been developed and shown strong generalization capabilities.
However, existing guardrails for LLMs, either ‘model guarding models’ ([, , ]) or ‘agent guarding models’ ([]), are designed for harmfulness defined on natural language.
They cannot be directly used to safeguard LLM agents with diverse output modalities.
In this paper, we propose GuardAgent, the first ‘agent guarding agents’ framework, and show its advantage over ‘model guarding agents’ approaches in our experiments.

## 3 Safety Requests for Diverse LLM Agents

Before introducing our GuardAgent, we investigate safety requests for different types of LLM agents in this section.
We focus on two representative LLM agents: an EHRAgent for healthcare and a web agent SeeAct.
In particular, EHRAgent represents LLM agents for high-stake tasks, while SeeAct represents generalist LLM agents for diverse tasks.
We briefly review these two agents, their designated tasks, and their original evaluation benchmarks.
More importantly, we propose two novel benchmarks for different safety requests: 1) EICU-AC, which assesses access control for healthcare agents like EHRAgent, and 2) Mind2Web-SC, which evaluates safety control for web agents like SeeAct.
Then, we conduct a preliminary study to test ‘invasive’ approaches for access control and safety control, which are based on naive instructions injected into the system prompts of EHRAgent and SeeAct, respectively; their ineffectiveness and poor generalization motivate the need for our GuardAgent.

Figure: Figure 2: An example from EICU-AC (left) and an example from Mind2Web-SC (right).
Refer to caption: extracted/5665097/figures/benchmark_example.png

### 3.1 EHRAgent and EICU-AC Benchmark

#### EHRAgent

EHRAgent is designed to respond to healthcare-related queries by generating code to retrieve and analyze data from provided databases [].
EHRAgent has been evaluated and shown decent performance on several benchmarks, including an EICU dataset containing questions regarding the clinical care of ICU patients (see Fig. 2 for example) and 10 relevant databases [].
Each database contains several types of patient information stored in different columns.
In practical healthcare systems, it is crucial to restrict access to specific databases based on user identities.
For example, personnel in general administration should not have access to patient diagnosis details.
Thus, LLM agents for healthcare, such as EHRAgent, should be able to deny requests for information from the patient diagnosis database when the user is from the general administration.
In essence, these LLM agents should incorporate access controls to safeguard patient privacy.

#### Proposed EICU-AC benchmark

In this paper, we create an EICU-AC benchmark from EICU to evaluate Access Control approaches for EHRAgent (and potentially other healthcare agents that require database retrieval).
We define three roles for the user of EHRAgent (and other similar target agents): ‘physician’, ‘nursing’, and ‘general administration’.
The access control being evaluated is supposed to ensure that each identity has access to only a subset of databases and columns of the EICU benchmark.
We generate the ground truth access permission for each role by querying ChatGPT (see App. A.1 for more details).
While generic access control approaches should be invariant to the specific roles and their access permissions, we have made these choices to simulate practical healthcare scenarios.
Then, each example in EICU-AC is designed to include the following information: 1) a healthcare-related question and the correct answer, 2) the databases and the columns required to answer the question, 3) a user identity/role, 4) a binary label ‘0’ if all required databases and columns are accessible to the given identity or ‘1’ otherwise, and 5) the required databases and columns inaccessible to the identity if the label is ‘1’.
An illustration of a generated EICU-AC example is shown in Fig. 2.

In particular, all questions in EICU-AC are sampled or adapted from the EICU dataset.
We keep questions from EICU that are correctly answered by EHRAgent using GPT-4 (at temperature zero) as the core LLM so that the evaluation using our benchmark will mainly focus on access control without much influence from the task performance.
Initially, we generate three EICU-AC examples from each of these questions by assigning them the three roles respectively.
After labeling each example based on the ground truth accessibility of its assigned role, we find for all three identities that the two labels are highly imbalanced.
Thus, for each identity, we remove some of the generated examples while adding new ones to achieve a relative balance between the two labels (see more details in App. A.2).
Ultimately, our EICU-AC contains 52, 57, and 45 examples labeled to ‘0’ for ‘physician’, ‘nursing’, and ‘general administration’, respectively, and 46, 55, and 61 examples labeled to ‘1’ for the three roles, respectively.
Moreover, among these 316 examples, there are 226 unique questions spanning 51 different ICU information categories, which underscores the diversity of our EICU-AC.

### 3.2 SeeAct and Mind2Web-SC Benchmark

#### SeeAct

SeeAct is a generalist web agent that follows natural language instructions to complete tasks on any given website by sequentially generating actions, including clicking on a button, typing specific texts, etc. (see Fig. 2 for example) [].
In the original paper, SeeAct is evaluated on the Mind2Web benchmark containing over 2,000 complex web tasks spanning 137 websites across 31 domains (e.g., car rental, shopping, entertainment, etc.) [].
However, it is essential for practical web agents like SeeAct to integrate safety controls that restrict certain actions for specific users.
For example, in most regions of the world, a driver’s license is required for car rental.

#### Proposed Mind2Web-SC benchmark

We create a Mind2Web-SC benchmark to evaluate Safety Control applicable to SeeAct and other web agents that operate based on action generation.
The objective of safety control is to ensure that the agent obeys six rules we created based on common web regulations and regional conventions: 1) user must be a member to shop, 2) unvaccinated user cannot book a flight, 3) user without a driver’s license cannot buy or rent a car, 4) user aged under 18 cannot book a hotel, 5) user must be in certain countries to search movies/music/video, 6) user under 15 cannot apply for jobs.
Again, these rules are proposed solely for evaluation purposes and do not reflect the personal views or attitudes of the authors.

The examples in Mind2Web-SC are created by the following steps.
First, we obtain all tasks with correct action prediction by SeeAct (using LLaVA-1.5 as the core LLM) from the travel, shop, and entertainment domains of the test set of Mind2Web.
Second, for each task, we randomly create a user profile containing ‘age’ in integer and ‘domestic’, ‘dr_license’, ‘vaccine’, and ‘membership’, all boolean (see the right of Fig. 2).
Note that each of these six user information categories is non-trivial, as it is related to at least one of the six safety rules we created.
Third, we manually label each example based on the task and the user information.
If the task itself is not related to any of the six rules, the example will be labeled to ‘0’ for ‘action permitted’.
If the task is related to at least one of the rules (e.g. the one for car rental), we check the user information and will label the example to ‘1’ for ‘action denied’ if the rule is violated (e.g. ‘dr_license’ is ‘false’) and ‘0’ otherwise.
For each example labeled to ‘1’, the violated rules are also included in our benchmark.
Finally, we balance the two classes by creating additional examples (based on existing tasks but with different user information) while removing some examples with tasks irrelevant to any of the rules (see details in App. B).
The created Mind2Web-SC benchmark contains 100 examples in each class with only unique tasks within the class.

**Table 1: Access control on EHRAgent and safety control on SeeAct based on system instructions are ineffective on EICU-AC and Mind2Web-SC; therefore, a new guardrail method is needed.**
|  | LPP $\uparrow$ | LPR $\uparrow$ | CCA $\uparrow$ | FRA $\uparrow$ |
| --- | --- | --- | --- | --- |
| EHRAgent + EICU-AC | 76.6 | 90.7 | 50.0 | 3.2 |
| SeeAct + Mind2Web-SC | 95.1 | 58.0 | 58.0 | 71.0 |

### 3.3 Preliminary Guardrails on the Two Benchmarks

#### Naive ‘invasive’ baselines

We use the EICU-AC benchmark to evaluate a naive approach that hardwires the access control into the system prompt of EHRAgent by specifying the three roles with their accessible databases and columns.
During the evaluation, this modified EHRAgent will be provided with both the role and the query of the EICU-AC examples.
Its system prompt will include instructions to display a ‘denial message’ along with the inaccessible databases and columns for the given role, if there are any.
Similarly, we incorporate textual instructions for safety checks into the system prompt of SeeAct and evaluate it on Mind2Web-SC.
If any of the rules are violated for the given user profile, the safety-enforced SeeAct is supposed to print a ‘denial message’ with the violated rules.
Details about the system prompts for the two agents equipped with the naive ‘invasive’ guardrails are deferred to App. C.

#### Metrics

We consider four evaluation metrics shared by both benchmarks: label prediction precision (LPP), label prediction recall (LPR), comprehensive control accuracy (CCA), and final response accuracy (FRA), all in percentage.
Both LPP and LPR are calculated over all examples in each dataset to measure the overall label prediction efficacy, where a prediction of label ‘1’ is counted only if the ‘denial message’ appears.
CCA considers all examples with ground truth labeled ‘1’.
It is defined as the percentage of these examples being correctly predicted to ‘1’ and with all inaccessible databases and columns (for EICU-AC) or all violated rules (for Mind2Web-SC) successfully detected.
In contrast, FRA considers all examples with ground truth labeled ‘0’.
It is defined as the percentage of these examples being correctly predicted to ‘0’ and with the agent responses correctly.

#### Results

As shown in Tab. 1, the two naive baselines fail in their designated tasks, exhibiting either low precision or recall in label prediction.
Specifically, the naive access control for EHRAgent is overly strict, resulting in an excessive number of false positives.
Conversely, the naive safety control for SeeAct fails to reject many unsafe actions, leading to numerous false negatives.
Moreover, the ‘invasion’ that introduces additional tasks imposes heavy burdens on both agents, significantly degrading the performance on their designated tasks, particularly for EHRAgent (which achieves only 3.2% end-to-end accuracy on negative examples as measured by FRA).
Finally, despite their poor performance, both naive guardrail approaches are hardwired to the agent, making them non-transferable to other LLM agents with different designs.
These shortcomings highlight the need for our GuardAgent, which is both effective and generalizable in safeguarding diverse LLM agents.

## 4 GuardAgent Framework

In this section, we introduce GuardAgent with three key features: 1) generalizable – the memory and toolbox of GuardAgent can be easily extended to address new target agents with new guard requests; 2) reliable – outputs of GuardAgent are obtained by successful code execution; 3) training-free – GuardAgent is in-context-learning-based and does not need any LLM training.

### 4.1 Overview of GuardAgent

The intended user of GuardAgent is the developer or administrator of a target LLM agent who seeks to implement guardrails on it.
The mandatory inputs to GuardAgent are all textual, including a set of guard requests $I_{r}$, a specification $I_{s}$ of the target agent, inputs $I_{i}$ to the target agent, and the output logs $I_{o}$ of the target agent corresponding to $I_{i}$.
Here, $I_{r}$ is informed by $I_{s}$, which includes the functionality of the target agent, the content of the inputs and output logs, their formats, etc.
The objective of GuardAgent is to check whether $I_{i}$ and $I_{o}$ satisfy the guard requests $I_{r}$ and then produce a label prediction $O_{l}$, where $O_{l}=0$ means the guard requests are satisfied and $O_{l}=1$ otherwise.
The outputs or actions proposed by the target agent will be admitted by GuardAgent if $O_{l}=0$ or denied if $O_{l}=1$.
If $O_{l}=1$, GuardAgent should also output the detailed reasons $O_{d}$ (e.g., by printing out the inaccessible databases and columns for EICU-AC) for potential further actions.
For example, severe rule violations for some use cases may require judicial intervention.

The key idea of GuardAgent is to leverage the logical reasoning capabilities of LLMs with knowledge retrieval to accurately ‘translate’ textual guard requests into executable code.
Correspondingly, the pipeline of GuardAgent comprises two major steps (see Fig. 1).
In the first step (Sec. 4.2), a step-by-step action plan is generated by prompting an LLM with the above-mentioned inputs to GuardAgent.
In the second step 4.3), we prompt the LLM with the action plan and a set of callable functions to get a guardrail code, which is then executed by calling an external engine.
A memory module is available in both steps to retrieve in-context demonstrations.

### 4.2 Task Planning

The objective for task planning is to generate a step-by-step action plan $P$ from the inputs to GuardAgent.
A naive design is to prompt a foundation LLM with $[I_{p},I_{s},I_{r},I_{i},I_{o}]$, where $I_{p}$ contains carefully designed planning instructions that 1) define each input to GuardAgent , 2) state the guardrail task (i.e., checking if $I_{r}$ is satisfied by $I_{i}$ and $I_{o}$), and 3) guide the generation of action steps (see Fig. 8 in App. D for a concrete example).
However, understanding the complex guard requests and incorporating them with the target agent remains a challenging task for existing LLMs.

We address this challenge by allowing GuardAgent to retrieve demonstrations from a memory module that archives target agent inputs and outputs from past use cases.
Here, an element $D$ in the memory module is denoted by $D=[I_{i,D},I_{o,D},P_{D},C_{D}]$, where $I_{i,D}$ and $I_{o,D}$ are the target agent inputs and outputs respectively, $P_{D}$ contains the action steps, and $C_{D}$ contains the guardrail code.
Retrieval is based on the similarity between the current target agent inputs and outputs and those from the memory.
Specifically, we retrieve $k$ demonstrations by selecting $k$ elements from the memory with the smallest Levenshtein distance $L([I_{i,D},I_{o,D}],[I_{i},I_{o}])$.
Then the action plan is obtained by $P={\rm LLM}([I_{p},I_{s},I_{r},[I_{i,D_{1}},I_{o,D_{1}},P_{D_{1}}],\cdots,[I_{
i,D_{k}},I_{o,D_{k}},P_{D_{k}}],I_{i},I_{o}])$.
Note that the guardrail code in each demonstration has been removed for the brevity of the prompt.

In the cases where GuardAgent is applied to a new LLM agent for some specific guard requests, we also allow the user of GuardAgent to manually inject demonstrations into the memory module.
In particular, we request the action plan in each demonstration provided by the user to contain four mandatory steps, denoted by $P_{D}=[p_{1,D},p_{2,D},p_{3,D},p_{4,D}]$, where the four steps form a chain-of-thought [].
In general, $p_{1,D}$ summarizes guard requests to identify the keywords, such as ‘access control’ with three roles, ‘physician’, ‘nursing’, and ‘general administration’ for EICU-AC.
Then, $p_{2,D}$ filters information in the guard request that is related to the target agent input, while $p_{3,D}$ summarizes the target agent output log and locates related content in the guard request.
Finally, $p_{4,D}$ instructs guardrail code generation to compare the information obtained in $p_{2,D}$ and $p_{3,D}$, as well as the supposed execution engine.
Example action plans are shown in Fig. 13 of App. G.

### 4.3 Guardrail Code Generation and Execution

The goal of this step is to generate a guardrail code $C$ based on the action plan $P$.
Once generated, $C$ is executed through the external engine $E$ specified in the action plan.
However, guardrail code generated by directly prompting an LLM with the action plan $P$ and straightforward instructions may not be reliably executable.
One of our key designs to address this issue is to adopt more comprehensive instructions that include a list $\mathcal{F}$ of callable functions with specification of their input arguments.
The definitions of these functions are stored in the toolbox of GuardAgent, which can be easily extended by users through code uploading to address new guard requests and target agents.
The LLM is instructed to use only the provided functions for code generation; otherwise, it easily makes up non-existent functions during code generation.

Furthermore, we utilize past examples retrieved from memory, employing the same approach used in task planning, to serve as demonstrations for code generation.
Thus, we have $C={\rm LLM}(I_{c}(\mathcal{F}),D_{1},\cdots,D_{k},I_{i},I_{o},P)$, where $I_{c}(\mathcal{F})$ are the instructions based on the callable functions in $\mathcal{F}$ and $D_{1},\cdots,D_{k}$ are the retrieved demonstrations.
The outputs of GuardAgent are obtained by executing the generated code, i.e., $(O_{l},O_{d})=E(C,\mathcal{F})$.
Finally, we adopt the debugging mechanism proposed by Shi et al. [], which invokes an LLM to analyze any error messages that may arise during execution to enhance the reliability of the generated code.

## 5 Experiments

In Sec. 5.2, we show the effectiveness of GuardAgent in safeguarding EHRAgent on EICU-AC and SeeAct on Mind2Web-SC with 98.7% and 90.0% label prediction accuracies, respectively.
We illustrate through a case study that the advantage of GuardAgent over ‘model guarding agents’ approaches is attributed to the more reliable guardrail by code generation and execution.
In Sec. 5.3, we conduct ablation studies to show 1) GuardAgent performs similarly well for most of the roles in EICU-AC and rules in Mind2Web-SC, allowing it to handle guard requests with high complexity, and 2) GuardAgent requires only a few shots of demonstrations.
In Sec. 5.4, we demonstrate that GuardAgent may define necessary functions based on guard requests, highlighting its ability to generalize to new guard requests.
Additionally, we find that LLMs, such as GPT-4, tend to generate code-based guardrails (albeit mostly inexecutable) even when not provided with specific instructions for code generation and execution.

### 5.1 Setup

#### Datasets and agents

We test GuardAgent on EICU-AC and Mind2Web-SC with EHRAgent and SeeAct (using their original settings) as the target agents, respectively.
The role and question from each EICU-AC example are inputs to EHRAgent, and the output logs include the reasoning steps, the generated code, and the final answer produced by EHRAgent.
The inputs to SeeAct contain the task and user information from each example in Mind2Web-SC, and the output logs include the predicted action and the reasoning by SeeAct.
Example inputs ($I_{i}$) and output logs ($I_{o}$) of the two target agents (which are the inputs to GuardAgent) are shown in App. D.
Other inputs to GuardAgent, including the specifications of the two target agents ($I_{s}$), the guard requests associated with the two benchmarks ($I_{r}$), and the planning instructions ($I_{p}$), are also shown in App. D due to space limitations.

#### Settings of GuardAgent

In the main experiments, we set the number of demonstrations to $k=1$ and $k=3$ for EICU-AC and Mind2Web-SC, respectively.
Other choices will be considered in our ablation study in Sec. 5.3.
For each benchmark, we initialize the memory of GuardAgent by $k$ manually created demonstrations (see App. G for example).
We use GPT-4 version 2024-02-01 with temperature zero as the core LLM of GuardAgent.
We use Python as the default code execution engine, with two initial functions in the toolbox, ‘CheckAccess’ and ‘CheckRules’, which are defined in App. E.
Note that users of GuardAgent can easily upload new functions or engines into the toolbox.
Finally, we allow three debugging iterations, though in most cases, the guardrail code generated by GuardAgent is directly executable.

#### Baselines

Since GuardAgent is the first LLM agent designed to safeguard other agents, we compare it with baselines using models to safeguard agents.
Here, we consider GPT-4 version 2024-02-01 and Llama3-70B as the guardrail models(^1^11Approaches for ‘model guarding models’, such as LlamaGuard designed to detect predefined unsafe categories [], are not considered here due to their completely different objectives.).
We create comprehensive prompts containing high-level instructions $I^{\prime}_{p}$ adapted from the one for GuardAgent, the same number of demonstrations as for GuardAgent but without guardrail code generation, denoted by $D^{\prime}_{1},\cdots,D^{\prime}_{k}$, and the same set of inputs as for GuardAgent.
However, neither baselines involve the memory module as our GuardAgent does; they use a fixed set of demonstrations during the evaluation.
Example prompts for both benchmarks are shown in App. H.
Then the outputs of the guardrail models for the baselines are obtained by $(O_{l},O_{d})={\rm LLM}(I^{\prime}_{p},I_{s},I_{r},D^{\prime}_{1},\cdots,D^{
\prime}_{k},I_{i},I_{o})$.

#### Evaluation metrics

We use the two label prediction metrics, LPP and LPR, and the CCA metric, all defined in Sec. 3.3.
The FRA metric is not considered here since all guardrails being evaluated will not affect the normal operation of the target agent when the alarm is not triggered.
In addition, we report the label prediction accuracy (LPA, a.k.a. guarding accuracy), defined over all examples in each dataset, as the overall metric for the guardrail performance.

**Table 2: Performance of GuardAgent in safeguarding EHRAgent on EICU-AC and SeeAct on Mind2Web-SC, compared with two model-based baselines with GPT-4 and Llama3, respectively.**
|  | EHRAgent on EICU-AC | SeeAct on Mind2Web-SC |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | LPA $\uparrow$ | LPP $\uparrow$ | LPR $\uparrow$ | CCA $\uparrow$ | LPA $\uparrow$ | LPP $\uparrow$ | LPR $\uparrow$ | CCA $\uparrow$ |
| Llama3 | 92.1 | 95.4 | 88.9 | 41.4 | 76.5 | 93.4 | 57.0 | 57.0 |
| GPT-4 | 97.5 | 95.3 | 100.0 | 67.9 | 82.5 | 100.0 | 65.0 | 65.0 |
| GuardAgent | 98.7 | 100.0 | 97.5 | 97.5 | 90.0 | 100.0 | 80.0 | 80.0 |

Figure: Figure 3: Left: A failure case of the GPT-4 baseline where the same column name (‘patientunitstayid’) shared by different databases cannot be effectively distinguished. Right: A failure case of GuardAgent where a rule violation is not detected due to the overwhelming details in the query.
Refer to caption: extracted/5665097/figures/case_study.png

### 5.2 Guardrail Performance

In Tab. 2, we show the performance of GuardAgent compared with the baselines using our comprehensive evaluation metrics.
GuardAgent achieves better LPAs than the two baselines with also clear gaps in CCAs, showing the advantage of ‘agent guarding agents’ over ‘model guarding agents’.
We attribute this advantage to our design of reasoning-based code generation and execution, which is clearly infeasible by guardrail models.
In many failure cases of GPT-4 on EICU-AC, we found that guardrails based on natural language cannot effectively distinguish column names if they are shared by different databases.
For example, in Fig. 3, the entire database ‘vitalperiodic’ that contains a column named ‘patientunitstayid’ is not accessible to ‘general administration’, while the column with the same name in the database ‘patient’ is accessible to the same role.
In this case, the model-based guardrail using GPT-4 fails to determine the column ‘patientunitstayid’ in the database ‘vitalperiodic’ as ‘inaccessible’.
In contrast, our GuardAgent based on code generation accurately converts each database and its columns into a dictionary, effectively avoiding such ambiguity in column names.

On the right of Fig. 3, we show a typical failure case of GuardAgent where the violated rule is undetected.
We found that the query failed to be connected to the designated rule in the first step of the chain-of-thought reasoning during task planning, possibly due to the overwhelming details in the query.
However, this issue can be mitigated by involving more demonstrations with better linguistic diversity, or using more powerful LLM as the core reasoning step.

**Table 3: Breakdown of GuardAgent results over the three roles in EICU-AC and the six rules in Mind2Web-SC. GuardAgent performs uniformly well for all roles and rules except for rule 5 related to movies, music, and videos.**
|  | EHRAgent on EICU-AC | SeeAct on Mind2Web-SC |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | physician | nursing | GA | rule 1 | rule 2 | rule 3 | rule 4 | rule 5 | rule 6 |
| LPA $\uparrow$ | 97.9 | 98.2 | 100.0 | 89.5 | 91.7 | 87.5 | 83.3 | 52.4 | 83.3 |
| CCA $\uparrow$ | 95.7 | 96.4 | 100.0 | 89.5 | 91.7 | 87.5 | 83.3 | 52.4 | 83.3 |

### 5.3 Ablation Studies

#### Breakdown results

In Tab. 3, we show LPA and CCA of GuardAgent for a) EHRAgent for each role of EICU-AC and b) SeeAct for each rule of EICU-AC (by only considering positive examples).
In general, GuardAgent performances uniformly well for the three roles in EICU-AC and the six rules in Mind2Web-SC except for rule 5 related to movies, music, and videos.
We find that all the failure cases for this rule are similar to the one illustrated in Fig. 3 where the query cannot be related to the rule during reasoning.
Still, GuardAgent demonstrates relatively strong capabilities in handling complex guard requests with high diversity.

#### Influence of number of demonstrations

We vary the number of demonstrations used by GuardAgent and show the corresponding LPAs and CCAs in Fig. 4.
The results show that GuardAgent can achieve descent guardrail performance with very few shots of demonstrations.

Figure: Figure 4: Performance of GuardAgent with different numbers of demonstrations on EICU-AC and Mind2Web-SC. GuardAgent is effective with very few demonstrations.
Refer to caption: extracted/5665097/figures/n_demo_eicu-ac.png

### 5.4 Code-Based Guardrail is the Natural Preference of LLMs, but Tools are Needed

We consider a challenging task where GuardAgent is instructed to generate guardrail code, but is provided with neither a) the functions needed for the guard requests nor b) demonstrations for guardrail code generation.
Specifically, the guardrail code is now generated by $C^{\prime}={\rm LLM}(I_{c}(\mathcal{F}^{\prime}),I_{i},I_{o},P)$, where $\mathcal{F}^{\prime}$ represents the toolbox without the required functions.
In this case, GuardAgent either defines the required functions or produces procedural code towards the same goal (see App. H for an example guardrail function generated by GuardAgent), and has achieved a 90.8% LPA with a 96.1% CCA on EICU-AC.
These results support the need for the list of callable functions and the demonstrations as our key design for the code generation step.
They also demonstrate a decent zero-shot generalization capability of GuardAgent to address new guard requests.

Moreover, we consider an even more challenging guardrail task.
We use the GPT-4 model to safeguard EHRAgent on EICU-AC, but remove all instructions related to code generation.
In other words, the LLM has to figure out its way, either with or without code generation, to provide a guardrail.
Interestingly, we find that for 68.0% examples in EICU-AC, the LLM chose to generate a code-based guardrail (though mostly inexecutable).
This result shows the intrinsic tendency of LLMs to utilize code as a structured and precise method for guardrail, supporting our design of GuardAgent based on code generation.

## 6 Conclusion

In this paper, we present the first study on guardrails for LLM agents to address diverse user safety requests.
We propose GuardAgent, the first LLM agent framework designed to safeguard other LLM agents.
GuardAgent leverages knowledge-enabled reasoning capabilities of LLMs to generate a task plan and convert it into a guardrail code.
It is featured by the generalization capabilities to new guardrail requests, the reliability of the code-based guardrail, and the low computational overhead.
In addition, we propose two benchmarks for evaluating privacy-related access control and safety control of LLM agents for healthcare and the web, respectively.
We show that GuardAgent outperforms ‘model guarding agent’ baselines on these two benchmarks and the code generalization capabilities of GuardAgent under zero-shot settings.

## Limitations

While GuardAgent performs well on the two benchmarks with also evidence of its generalization capabilities, it requires the core LLM to have descent reasoning capabilities.
This limitation is due to the complexity of both the guardrail tasks and the target agent to be safeguarded.
However, this limitation can be mitigated as current LLMs are becoming more and more powerful in reasoning.

## Broader Impacts

We propose GuardAgent with potentially positive social impacts.
GuardAgent is the first LLM agent framework that safeguards other LLM agents.
GuardAgent directly addresses the safety and trustworthiness concerns of LLM agents and will potentially inspire more advanced guardrail approaches for LLM agents.

## Appendix A Details About the EICU-AC Benchmark

### A.1 Role-Based Access Permission

For the EICU-AC benchmark, we consider three roles: ‘physician’, ‘nursing’, and ‘general administration’.
These roles are selected based on our understanding of the ICU environment.
Although various other roles exist, we focus on these three roles due to their prevalence, ensuring sufficient queries relevant to each role when creating the benchmark.

Figure: (a) List of all databases and columns.
Refer to caption: extracted/5665097/figures/access_all.png

For each role, we select a subset of accessible databases and columns from the EICU benchmark, as shown in Fig. 5.
Our selection rule is to query ChatGPT about the access permission for the three roles over each database.
For example, for the ‘diagnosis’ database with four columns, ‘patientunitstayid’, ‘icd9code’, ‘diagnosisname’, and ‘diagnosistime’, we query ChatGPT using the prompt shown in Fig. 6.
ChatGPT responds with the recommended access permission (‘full access’, ‘limited access’, or ‘no access’) for each role to each of the four columns.
Here, we follow all ‘full access’ and ‘no access’ recommendations by ChatGPT.
For ‘limited access’, we set it to ‘no access’ if it is recommended for ‘physician’ or ‘nursing’; if it is recommended for ‘general administration’, we set it to ‘full access’.
This is to ensure both ‘physician’ and ‘nursing’ roles have sufficient inaccessible databases so that there will be sufficient queries that should be denied in the ground truth (to achieve relatively balanced labeling for both roles).

Figure: Figure 6: Our prompt to ChatGPT for the access permission for the three roles to the ‘diagnosis’ database (with four columns, ‘patientunitstayid’, ‘icd9code’, ‘diagnosisname’, and ‘diagnosistime’), and the responses of ChatGPT.
Refer to caption: extracted/5665097/figures/access_query.png

### A.2 Sampling from EICU

As mentioned in the main paper, each example in EICU-AC contains 1) a healthcare-related question and the correct answer, 2) the databases and the columns required to answer the question, 3) a user identity, 4) a binary label (either ‘0’ for ‘access granted’ and ‘1’ for ‘access denied’), and 5) databases and the columns required to answer the question but not accessible for the given role (if there are any).
The examples in EICU-AC are created by sampling from the original EICU dataset following the steps below.
First, from the 580 test examples in EICU, we obtain 183 examples that are correctly responded to by EHRAgent with GPT-4 at temperature zero.
For each of these examples, we manually check the code generated by EHRAgent to obtain the databases and columns required to answer the question.
Second, we assign the three roles to each example, which gives 549 examples in total.
We label these examples by checking if any of the required databases or columns are inaccessible to the given role (i.e., by comparing with the access permission for each role in Fig. 5).
This will lead to a highly imbalanced dataset with 136, 110, and 48 examples labeled ‘0’ for ‘physician’, ‘nursing’, and ‘general administration’, respectively, and 47, 73, and 135 examples labeled ‘1’ for ‘physician’, ‘nursing’, and ‘general administration’, respectively.
In the third step, we remove some of the 549 created examples to a) achieve a better balance between the labels and b) reduce the duplication of questions among these examples.
We notice that for ‘general administration’, there are many more examples labeled ‘1’ than ‘0’, while for the other two roles, there are many more examples labeled ‘0’ than ‘1’.
Thus, for each example with ‘general administration’ and label ‘1’, we remove it if any of the two examples with the same question for the other two roles are labeled ‘1’.
Then, for each example with ‘nursing’ and label ‘1’, we remove it if any example with the same question for ‘physician’ is labeled ‘1’.
Similarly, we remove each example with ‘physician’ and label ‘0’ if any of the two examples with the same question for the other two roles are also labeled ‘0’.
Then for each example with ‘nursing’ and label ‘0’, we remove it if any example with the same question for ‘general administration’ is labeled ‘0’.
After this step, we have 41, 78, and 48 examples labeled ‘0’ for ‘physician’, ‘nursing’, and ‘general administration’, respectively, and 47, 41, and 62 examples labeled ‘1’ for ‘physician’, ‘nursing’, and ‘general administration’, respectively.
Finally, we randomly remove some examples for ‘nursing’ with label ‘0’ and ‘general administration’ with label ‘1’, and randomly add some examples for the other four categories (‘physician’ with label ‘0’, ‘general administration’ with label ‘0’, ‘physician’ with label ‘1’, and ‘nursing’ with label ‘1’) to achieve a better balance.
The added examples are generated based on the questions from the training set(^2^22In the original EICU dataset, both the training set and the test set do not contain the ground truth answer for each question. The ground truth answers in the test set of EICU are provided by Shi et al. [].) of the original EICU benchmark.
The ultimate number of examples in our created EICU-AC benchmark is 316, with the distribution of examples across the three roles and two labels displayed in Tab 4.

**Table 4: Number of examples in EICU-AC for each role and each label.**
|  | physician | nursing | general administration |
| --- | --- | --- | --- |
| label ‘0’ (access denied) | 52 | 57 | 45 |
| label ‘1’ (access granted) | 46 | 55 | 61 |

### A.3 Healthcare Questions Involved in EICU-AC

As mentioned in the main paper, our created EICU-AC dataset involves healthcare questions spanning 50 different ICU information categories, i.e., columns across all 10 databases of the EICU benchmark.
We further categorize the questions in EICU-AC following the ‘template’ provided by EICU (extracted from the ‘q_tag’ entry of each example []).
This gives 70 different question templates, showing the high diversity of healthcare questions involved in our EICU-AC benchmark.

**Table 5: Number of examples labeled ‘1’ in Mind2Web-SC for each rule violation. Note that examples labeled ‘0’ do not violate any rules.**
| Safety rules | No. examples |
| --- | --- |
| Rule 1: User must be a member to shop. | 19 |
| Rule 2: Unvaccinated user cannot book a flight | 12 |
| Rule 3: User without a driver’s license cannot buy or rent a car. | 24 |
| Rule 4: User aged under 18 cannot book a hotel. | 18 |
| Rule 5: User must be in certain countries to search movies/musics/video. | 21 |
| Rule 6: User under 15 cannot apply for jobs. | 6 |

## Appendix B Details About the Mind2Web-SC Benchmark

In Sec. 3.2, we have defined six safety rules for the Mind2Web-SC Benchmark.
Rule 1 requires ‘membership’ in the user information to be ‘true’.
Rule 2 requires ‘vaccine’ in the user information to be ‘true’.
Rule 3 requires ‘dr_license’ in the user information to be ‘true’.
Rule 4 requires ‘age’ in the user information to be no less than 18.
Rule 5 requires ‘domestic’ in the user information to be ‘true’.
Rule 6 requires ‘age’ in the user information to be no less than 15.
In Tab. 5, we show the number of examples labeled ‘1’ in Mind2Web-SC for each rule violation.
Note that examples labeled ‘0’ do not violate any rules.

During the construction of Mind2Web-SC, we added some examples with label ‘1’ and removed some examples with label ‘0’ to balance the two classes.
By only following the steps in Sec. 3.2 without any adding or removal of examples, we obtain a highly imbalanced dataset with 178 examples labeled ‘0’ and only 70 examples labeled ‘1’.
Among the 178 examples labeled ‘0’, there are 148 examples with the tasks irrelevant to any of the rules – we keep 50 of them and remove the other $(148-50=)$ 98 examples.
All 30 examples labeled ‘0’ but related to at least one rule are also kept.
Then, we create 30 examples labeled ‘1’ by reusing the tasks for these 30 examples labeled ‘0’.
We keep generating random user profiles for these tasks until the task-related rule is violated, and the example is labeled to ‘1’.
Note that the tasks are randomly selected but manually controlled to avoid duplicated tasks within one class.
Similarly, we created 20 examples labeled ‘0’ by reusing the tasks for examples labeled ‘1’, with randomly generated user information without any rule violation.
Finally, we obtain the Mind2Web-SC dataset with 100 examples in each class (200 examples in total).
Among the 100 examples labeled ‘0’, 50 are related to at least one of the rules.

## Appendix C Detailed System Prompts for Naive Access Control and Safety Control Based on Instructions

In our preliminary studies, We created a naive access control for EHRAgent and a naive safety control for SeeAct by directly modifying their system prompts for planning.
These approaches are either ineffective in safeguarding the agents or degrade the benign performance of the agents.
In Fig. 7, we show the instructions we injected into the system prompts of these two agents.

Figure: Figure 7: Instructions injected into the system prompt of EHRAgent for access control and SeeAct for safety control, as naive baselines that motivate our GuardAgent.
Refer to caption: extracted/5665097/figures/prompt_baseline_naive.png

## Appendix D Complete Inputs and Output of GuardAgent

#### Complete Inputs to GuardAgent

As described in Sec. 4.2, the inputs to GuardAgent include a specification $I_{s}$ of the target agent, a set of guard requests $I_{r}$, inputs $I_{i}$ to the target agent, and the output log $I_{o}$ by the target agent corresponding to $I_{i}$.
In Fig. 8, we show the actual $I_{s}$ and $I_{r}$ for GuardAgent in our experiments for both EHRAgent on EICU-AC and SeeAct on Mind2Web.
In the same figure, we also show the actual planning instruction $I_{p}$ and the actual instruction $I_{c}$ used for code generation.
In Fig. 9, we show real examples for the target agent inputs $I_{i}$ and output logs $I_{o}$ for both EHRAgent on EICU-AC and SeeAct on Mind2Web.

Figure: Figure 8: The actual planning instruction $I_{p}$, instruction $I_{c}$ for guardrail code generation, target agent specification $I_{s}$ and guard requests $I_{r}$ we used in our experiments for the two agents, EHRAgent and SeeAct, and the two benchmarks, EICU-AC and Mind2Web-SC.
Refer to caption: extracted/5665097/figures/input_system_guardagent.png

Figure: Figure 9: Examples for target agent inputs $I_{i}$ and output logs $I_{o}$, as the inputs to GuardAgent, for the two agents, EHRAgent and SeeAct, and the two benchmarks, EICU-AC and Mind2Web-SC.
Refer to caption: extracted/5665097/figures/input_data_guardagent.png

#### Outputs of GuardAgent

The intermediate outputs of GuardAgent, including the generated action plan $P$ and the guardrail code $C$, are similar to those in the manually created demonstrations (see App. G); thus, we do not repeatedly show them.
Here, we show example outputs, including the label prediction $O_{l}$ and the detailed reasons $O_{d}$ of GuardAgent for both benchmarks in Fig. 10.

Figure: Figure 10: Example outputs of GuardAgent, including the label prediction $O_{l}$, the detailed reasons $O_{d}$, and the final answer/action of the target agent with guardrail, for the two agents, EHRAgent and SeeAct, and the two benchmarks, EICU-AC and Mind2Web-SC.
Refer to caption: extracted/5665097/figures/output_guardagent.png

## Appendix E Callable Functions

Our experiments involve two callable functions shown in Fig. 11.
However, the user of GuardAgent can easily extend the toolbox by uploading more functions.
The actual toolbox of GuardAgent in practice will contain much more callable functions than for our experiments.

Figure: Figure 11: Callable functions in the toolbox of GuardAgent involved in our experiments.
Refer to caption: extracted/5665097/figures/functions.png

## Appendix F Prompts for Baselines

In the main experiments, we compare GuardAgent with two baselines using LLMs to safeguard LLM agents.
The guardrail is created by prompting the LLM with a system instruction, the specification of the target agent, the guard requests, the user inputs to the target agent with the associated output logs, and a few show of examples.
Here the system instruction is adapted from the one used by GuardAgent for task planning.
However, we include additional instructions about the format of the guardrail outputs.
The baselines do not involve any guardrail code generation, and this is reflected by the demonstrations we created that generate guardrails solely based on reasoning over the textual inputs to the LLM.
In Fig. 12, we show the modified system prompt template for the baselines, with two example demonstrations for the two benchmarks, respectively.

Figure: Figure 12: System prompt template for the baselines and the two example demonstrations for EICU-AC and Mind2Web-SC, respectively.
Refer to caption: extracted/5665097/figures/prompt_baseline.png

## Appendix G Manually Created Demonstrations

We manually created a set of demonstrations for each benchmark.
In Fig. 13, we show two example demonstrations for EHRAgent on EICU-AC and SeeAct on Mind2Web-SC, respectively.

Figure: Figure 13: Example demonstrations for EHRAgent on EICU-AC and SeeAct on Mind2Web-SC.
Refer to caption: extracted/5665097/figures/example_demos.png

## Appendix H Function Defined by GuardAgent in Zero-Shot Setting

In the zero-shot setting where GuardAgent is provided with neither the required functions nor demonstrations for guardrail code generation, GuardAgent can still generate guardrails by defining new functions.
In Fig. 14, we show a function defined by GuardAgent during guardrail code generation.
The function differs from those we provided in Fig. 11, but it achieves the same guardrail goals.

Figure: Figure 14: A function defined by GuardAgent in zero-shot setting with neither demonstrations for code generation nor required functions
Refer to caption: extracted/5665097/figures/codegen.png

## Appendix I Execution Time of GuardAgent

The average execution time for GuardAgent (with GPT-4) safeguarding EHRAgent on EICU-AC is 45.4 seconds per example, while the average execution time for EHRAgent (with GPT-4) is 31.9 seconds per example.
The average execution time for GuardAgent (with GPT-4) safeguarding SeeAct on Mind2Web-SC is about 60 seconds per example, while the average execution time for EHRAgent (with LLaVA-1.5) is about 20 seconds per example.
In general, the execution time for GuardAgent is comparable to the execution time of the target agent.
Moreover, human inspectors will likely need much more time than our GuardAgent to read the guard requests and then moderate the inputs and outputs of the target agent correspondingly.
Given the effectiveness of our GuardAgent as shown in the experiments, we believe that GuardAgent is the current best for safeguarding LLM agents.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. Could you clarify which specific work is being referred to as "agent guarding models" approach？
2. Additionally, it appears that related work on "model guarding agents" approaches has not been cited, right?
3. Could you discuss more about current limitations/challenges and future direction of “agent guarding agent” approach?
soundness: 2
strengths: • The paper is well-structured, with a clear logical flow that facilitates understanding. The experimental design is concise, and the results are presented in an easily interpretable manner, enhancing the clarity of the study.

• The paper effectively underscores the necessity of the proposed "agent guarding agent" approach, particularly highlighting GuardAgent's importance in accommodating dynamic and complex guardrail requests. This emphasis supports the relevance and timeliness of the guardrail framework in addressing complex safety and privacy challenges.
summary: This paper proposes GuardAgent, the first LLM agent designed to safeguard other LLM agents. GuardAgent utilizes the reasoning capabilities of LLMs to generate a task plan and translate it into guardrail code. It stands out for its flexibility in handling diverse guardrail requests by retrieving relevant demonstrations from a memory module, its reliability through code-based guardrails, and its low computational overhead, requiring no additional LLM training.
weaknesses: • Lack of experimental comparison with existing works on Guardrail methods. While the related work section discusses existing many guardrail approaches, the study conducts only a brief comparison with “model guarding agent” approach. Expanding the scope of comparison with a broader range of guardrail techniques would strengthen the evaluation of GuardAgent’s effectiveness.

• In the ablation studies, the authors mention “the trend of code-based guardrails” as a rationale for the code-generation design of GuardAgent, but this observation has only been briefly mentioned. This aspect appears intriguing, and further experimental analysis and a more detailed discussion would enhance the understanding of this design choice.

• As the first work to explore “agent guarding agents”, this paper is positioned to serve as a key reference for future research in this domain. However, to support subsequent studies, it would benefit from an analysis of the GuardAgent limitations or potential threats to validity associated with current version of paper, as well as a discussion of possible future directions on “agent guarding agent”  to further advance the development of robust guardrail frameworks.

• Some typos are found.
- Section 4.1: In the second step 4.3),-> In the second step (Sec. 4.3)
- Why all metrics in all Tables has upwards arrow?
- Section 5.3: Tab. 2-> Table 2, Fig. 3-> Figure 3
- Please give a formal(short) caption for all Figures and Tables

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: **Q1**: It is better to briefly introduce the definition of ‘model-guard-agent’ baseline when it first appears in this paper. Otherwise, it will confuse the readers.
soundness: 3
strengths: **S1**: A valuable contribution of this paper is providing the LLM community with the probably first safety benchmarks that explicitly model user profiles. In these two benchmarks, the actions that the LLM agent needs to perform are related to the user's identity and profile, requiring the agent to assess whether the user's actions may pose potential risks based on their identity. These datasets present a greater challenge for LLM agents, as they need to incorporate the user profile and assigned permissions into the context to complete tasks while minimizing risks. Such benchmarks closely align with real-world needs, specifically in providing different services based on varying user permissions. This approach offers the LLM community a new perspective and dimension, while also presenting new challenges.

**S2**: In tasks aimed at enhancing the alignment or safety capabilities of LLMs, safety and helpfulness are often in conflict, requiring a trade-off. Specifically, as the safety capabilities of LLMs increase, the likelihood of them generating refusal responses also increases, which in turn reduces the likelihood of providing helpful information to the user—a phenomenon known as the "safety tax." However, in this paper, experiments demonstrate that GuardAgent does not affect the task performance of the target agent while safeguarding against potential risks. This design represents a well-executed approach.
summary: This paper presents GuardAgent, a protective guardrail framework that functions as a third-party safeguard for other LLM agents. GuardAgent initially utilizes an LLM to develop an action plan derived from guard requests as well as the inputs and outputs of the target agent. The LLM then transforms this plan into guardrail code, which is subsequently executed by an external engine. The authors also introduce two benchmarks specifically aimed at evaluating LLM safety: EICU-AC, designed to test access control for LLM agents in healthcare, and Mind2Web-SC, a dataset intended to assess safety mechanisms for web agents powered by LLMs. Experimental validation on these datasets shows that GuardAgent performs better than the "model-guard-agent" baseline, which uses hard-coded task instructions.
weaknesses: **W1**: The work in this paper first designs two benchmarks, i.e., EICU-AC and Mind2Web-SC, and then develops the method based on these two tasks. However, the motivation for proposing these benchmarks has not been well justified. Why are database retrieval (EICU-AC) and web service calls (Mind2Web-SC) the two representative tasks for testing the safety of LLM agents? What makes these two tasks sufficiently representative to measure the safety capabilities of LLM agents in mitigating risks? Are there any similar tasks in previous research that have been used to validate the safety of LLM agents? Why introduce these new tasks instead of using existing benchmarks? In summary, there needs to be an explanation and justification for the motivation behind proposing these new benchmarks.

**W2**: At the beginning of this paper, one contribution is introduced as ``generate guardrail code based on the task plan." However, after reading the benchmark design section, we find that the so-called guardrail code generation is actually a task-oriented function, not a task-agnostic, universally applicable design. Since the EICU-AC task itself requires generating structured query code for database retrieval, GuardAgent needs to have code generation capabilities. This doesn’t mean that a unique code design was developed to enhance the safety of LLM agents. Here, if the downstream task doesn’t require code generation, such as in a complex Q&A task where responses are given in natural language, would code generation then be a redundant design for GuardAgent? 

**W3**: Additionally, the authors state that GuardAgent leverages the LLM's reasoning capabilities to "accurately ‘translate’ textual guard requests into executable code." However, this capability is task-dependent. When the task does not require code generation, wouldn’t this reasoning ability be unnecessary? Therefore, the contribution proposed in this paper—particularly the guardrail code generation—is determined by the characteristics of the downstream tasks. It is not inherently a design that can be applied to various tasks to enhance LLM agent safety, which weakens this work's generalizability and impact.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: See the Weaknesses above.
soundness: 3
strengths: 1. GuardAgent is the first framework focused on providing guardrails to LLM agents, addressing a critical gap in AI agent safety and privacy.

2. The system's non-invasive approach and extendable toolbox make it adaptable to diverse LLM agents and new guard requests.

3. GuardAgent demonstrates superior accuracy and reliability compared to baseline models, particularly when using code-based guardrails.
summary: This paper presents GuardAgent, a framework designed to enhance the safety and robustness of LLMs against adversarial inputs and potential misuse. GuardAgent leverages a multi-agent architecture, incorporating safety-checking agents that detect unsafe responses, and employs prompt engineering to guide the LLM toward safer outputs. The framework integrates several defensive techniques, including uncertainty estimation, self-refinement prompts, and cross-agent validation, to achieve a high level of safety without compromising the model’s performance. In addition to GuardAgent, this paper also proposes two novel benchmarks: an EICU-AC benchmark for assessing privacy-related access control for healthcare agents and a Mind2Web-SC benchmark for assessing safety regulations for web agents.
weaknesses: 1. **Dataset Scope**: The paper evaluates GuardAgent on two specific datasets, EICU-AC for privacy-related access control in healthcare and Mind2Web-SC for safety compliance in web agents. While these datasets represent different types of guard requests, the limited scope raises questions about GuardAgent's generalizability. As a defense mechanism, how effectively can GuardAgent adapt to other domains beyond healthcare and web safety, where guardrails may vary significantly?

2. **Performance Variability**: The paper evaluates GuardAgent using different core LLMs, such as Llama3-70B and GPT-4, but it does not fully clarify how this choice impacts GuardAgent’s overall effectiveness. Does GuardAgent’s performance, including its accuracy in identifying violations, speed in generating guardrail code, and flexibility in adapting to complex guard requests, vary significantly depending on the model used?

3. **Memory Dependency**: The ablation study shows that GuardAgent’s performance improves with more in-context demonstrations. How does the quality of these demonstrations, such as their relevance or diversity, impact accuracy? Is there an optimal number of demonstrations that balances accuracy and efficiency, and does this vary by task or application?

4. **Computational Efficiency**: What are the computational costs associated with GuardAgent’s code generation and execution process? 

5. **Error Handling**: GuardAgent includes a debugging mechanism that uses an LLM to analyze and address errors during code execution. Could you clarify how robust this mechanism is? Specifically, how well does it handle different error types (e.g., syntax errors, logical errors, unforeseen inputs), and are there limits to the errors it can reliably resolve? Additionally, has it been tested in scenarios with complex or ambiguous errors, and what are its typical failure modes, if any?

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: (1) Can the authors provide examples or comment on the ability of the framework to adapt to more complicated policies than the ones evaluated in the experimental section (such as more fine-grained access controls for different users)?

(2) Can the authors comment on the ability of GuardAgent to handle nonbinary requirements (e.g., safety requirements that work on a threshold, risk-based requirements, etc.)? 

(3) Can the authors comment on or provide evidence that the framework's code generation process is reliable and generalizable (i.e., generates low rates of inexecutable code even for new safety or privacy policies or application domains)? Under what scenarios or tasks might this change; do new policies that have limited to no occurrence in the memory module result in more inexecutables?

(4) Do the authors have an idea about why the recall LPR does better for GPT-4 baseline compared to GuardAgent (Table 2)?
soundness: 4
strengths: This paper presents an innovative framework for a LLM agent to safeguard other LLM agents. The design of the framework is noninvasive, generates guardrails using python code execution (not natural language) and does not require LLM training or retraining, which are all strengths. Using python code execution as opposed to some knowledge-specific languages as done in some other works (e.g., Kang, Li R^2-Guard) is a bonus, supporting adoption by a wider set of programmers/engineers. 

In addition, the integration of the memory module is a nice feature which allows the agent to document new cases it runs into and learn from previous examples which can help reduce the burden of a user needing to specify an exhaustive list of requirements, and may allow for adaption to new (unseen) scenarios. This is an innovation compared to previous work (Rebedea et al. NeMo Guardrails; Inan et al, Llama Guard; Ghosh et al. Aegis) which either require users to specify and provision every safety/privacy property or need some type of training guideline, such as classification labels. The paper evaluates the framework based on the curated benchmarks and compared to baselines for different LLM model types. The experiments make sense as they test precision, recall and two measures of accuracy for the labels about whether a policy is violated or not. The eval includes an ablation study which shows performance improvements for the inclusion of the memory module and toolbox (supporting framework architecture decisions), and provide evidence that the framework performs well with performance improvements compared to the baselines in almost all criteria.

Finally, this paper is presented very nicely- the structure, organization and language are very clear and the figures are unambiguous and helpful.
summary: This paper focuses on guardrails for LLM agents, with the authors proposing GuardAgent, a LLM agent designed to safeguard other LLM agents based on specified guard requests from users. The paper has the following contributions: (1) development of an LLM agent framework to guard other target LLM agents based on user requirements and using a memory module that stores previous use cases; (2) creation of two benchmarks for healthcare and web agents to evaluate access control policies; (3) evaluation and ablation study evaluating GuardAgent vs. baseline models.
weaknesses: The LLM guardrail space is quite saturated with recent work, and the related work section is quite concise. It might be worth expanding this section to better motivate the need for this work. For example, the authors mention that model- and agent- guarding cannot be directly used to safeguard  LLM agents with diverse output modalities. Can the authors provide some contextual examples or explain why?

The access control policies used for the healthcare and web agent scenarios and evaluation (though relevant for these application areas) are pretty simplistic (see question 1)). In many real world use cases, more complicated policies may be required to accurately provision safety/privacy constraints. 

The GuardAgent only returns binary (policy violated or not) responses, which may not be sufficient to represent all cases of safety and privacy policies. Many policies are not so strict, and may need to return risk-based or threshold based responses (see question 2).

A pro of the framework is the automatic code generation through use of the framework's toolbox. I am wondering about the reliability and generalizability of such a process as inexecutable generated code would render the framework useless (see question 3). This is not directly evaluated in the experimental section (though the authors mention code is almost always executable and does not often use the debugging stopgap in Section 4.3). 

It would be great if the authors intended to release their framework and benchmarks (e.g., so that people could contribute to the toolbox functionalities or specifications).

Meta-review: This paper presents GuardAgent - an "agent guarding agents" approach, as opposed to "models guarding agents". 
The code-based guardrail is designed to be more robust to accomodate guardrail requests, in comparison to natural-language based guardrails. GuardAgent responds to an explicit list of guardrail requests specified by the users. 
The method is evaluated on two tasks and datasets: database retrieval (EICU-AC) and web service calls (Mind2Web-SC) as the two representative tasks for testing the safety of the specific agents: medical agents and web agents.
The guardrail code generation is thus task-specific to the datasets and chosen benchmark tasks.
The authors created a set of safety rules for each dataset a original datasets did not include any rules - to test the QA tasks.
Overall the paper is well written and the experiments part are clear.

However, it is unclear still why the authors did not use well established QA tasks used for LLM safety, since they also introduced safety rules for the original datasets. This is a question by reviewer 4xoT and the responses by authors seem to have not fully addressed the questions on the specificity of the tasks, and in other scenarios when safety guardrails are not explicitly stated, or in other Q&A datasets/ benchmark tasks. The fact that the paper only is evaluated on two very specific datasets and agentic tasks is the major limiting factor. 

The authors were very responsive during the rebuttal period which is highly commendable.
Although two of the reviewers increased their score, this paper still falls exactly on the borderline mark. Although there is a champion reviewer, two of the reviewers have score 5, and with one of the active reviewers and expert is adamant that the paper has not quite passed the borderline. 

Therefore, I am recommending rejection for this paper and suggest the authors to polish this paper further for future submission.

Decision: Reject