Safe Reinforcement Learning in Black-Box Environments via Adaptive Shielding

## Abstract

Abstract Empowering safe exploration of reinforcement learning (RL) agents during training is a critical impediment towards deploying RL agents in many real-world scenarios. Training RL agents in unknown, black-box environments poses an even greater safety risk when prior knowledge of the domain/task is unavailable. We introduce ADVICE (Adaptive Shielding with a Contrastive Autoencoder), a novel post-shielding technique that distinguishes safe and unsafe features of state-action pairs during training, thus protecting the RL agent from executing actions that yield potentially hazardous outcomes. Our comprehensive experimental evaluation against state-of-the-art safe RL exploration techniques demonstrates how ADVICE can significantly reduce safety violations during training while maintaining a competitive outcome reward.

## 1 Introduction

Reinforcement Learning (RL) [] is a powerful machine learning paradigm for solving complex decision-making tasks that has exhibited performance commensurate with the cognitive abilities of humans in diverse applications, including game-playing [, , ] and robot control [, ]. Despite this huge potential, developing RL-based agents that can explore their environment safely remains a significant challenge. Exploring unfamiliar, and potentially hazardous, states while learning from the environment, especially in safety-critical domains, like robotics or healthcare, can pose real dangers. Alleviating this entails RL agents capable of synthesising an optimal policy by exploring the policy space adequately while ensuring safe exploration by preventing the execution of unsafe actions [].

Ensuring safety becomes an increasingly difficult challenge in complex environments typically characterised by high-dimensional state/action spaces []. Such environments require a large amount of training time before the agent can consistently complete the task and avoid safety concerns. This issue is further exacerbated in black-box environments where no prior knowledge can be utilised before training; the only information available is the data observed in real time by the RL agent. In such scenarios, the risk associated with exploration increases exponentially as the agent must operate without pre-defined guidelines, rendering typical safe exploration techniques inadequate [].

Prior research on safe RL exploration formulates safety constraints as linear temporal [, , ] and probabilistic logic [] specifications, whose use as a shield protects the agent during training. Shielding techniques are categorised into pre-shielding (restricting action choices to a predefined safe subset) and post-shielding (evaluating and modifying actions post-selection to ensure safety) []. Despite noteworthy advances, a common challenge across these methods is their reliance on some degree of prior knowledge about the environment, task, or safety concern which may not be generally available. Research targeting safe exploration in black-box environments employs Lagrangian methods [, , , ], or involves a pre-training phase before the shield synthesis [].

We present ADVICE (ADaptiVe ShIelding with a Contrastive AutoEncoder), a novel post-shielding technique for the safe exploration of RL agents in black-box environments. The ADVICE shield is underpinned by a contrastive autoencoder that effectively learns distinguishing latent representations between safe and unsafe features (state-action pairs) and a non-parametric classifier that employs these latent representations, allowing new features to be identified as safe, or correcting them with a new safe action when deemed unsafe. Further, ADVICE encompasses an adaptation component that considers the agent’s recent performance to automatically regulate the risk tolerance levels of the shield, thus encouraging exploration when appropriate. We demonstrate ADVICE’s ability to work in highly complex black-box environments and significantly reduce safety violations during training in comparison to state-of-the-art methods, such as Lagrangian multipliers [] and discretized shields [].
To the best of our knowledge, ADVICE is the first research work that investigates shielding for safe RL exploration in black-box environments with high-dimensional state/action spaces and proposes an end-to-end methodology for shield synthesis without using *any* prior knowledge.

## 2 Related Work

Shielding Techniques.
Recent research devises techniques for the safe exploration of RL agents using safety shields, allowing the agent to select from a pool of safe actions or correct an action deemed unsafe []. Existing shielding approaches leverage linear temporal logic (LTL) specifications [, , ] or use external hints for constructing the LTL formulae []. LTL specifications can be replaced with probabilistic logic programming (PLP) [], extending their use to continuous deep RL and enabling safety constraints to be differentiable. By utilising logical neural networks [], the same logic specifications can be both respected and learnt, providing a more nuanced understanding of safety. A recurring limitation of these methods is their reliance on explicit prior knowledge of their environment, task, and/or safety concerns. Although they can improve safety and, in some cases, eliminate violations altogether, their applicability is restricted to a narrow set of environments and safety considerations []. In contrast, our ADVICE post-shielding method does not need *any* prior knowledge, using exclusively the information captured in a typical RL problem.

Black-Box Safe Exploration Techniques. Other recent research focuses on improving safety in black-box environments, where no prior knowledge is provided to the agent/user. A trivial but effective solution is to record all unsafe features in a tabular format to prevent the agent from repeating them []. However, this approach is limited to discrete environments or extremely low-dimensional spaces. Other research collects data in the environment before training, to then utilise a safety layer [, , ] or shield [] to protect the agent. This requires a significant amount of data collection, resulting in an increasing number of safety violations. Lagrangian methods enable modelling the Markov Decision Process as a Constrained Markov Decision Process, leading to their wide adoption due to their simplicity and effectiveness [, ]. The Lagrangian multiplier can be fixed [, ], or integrated into the algorithm itself [, ]. Other advancements employ uncertainty estimation concepts [, ]. Defining safety in terms of uncertainty and propagating it into the RL algorithm during training yields a cautious yet effective agent for reducing safety violations, even in complex environments. Similarly to these approaches, ADVICE needs no prior knowledge; however, ADVICE also entails far less data collection than the previously discussed techniques.

## 3 Preliminaries

Markov Decision Process.
A Markov Decision Process (MDP) [] is a discrete-time stochastic control process to model decision-making. An MDP is formally defined as a 5-tuple $M=(S,A,P,R,\gamma)$, where $S$ is the state space, $A$ is the action space, $P$ is the state transition probability matrix such that $P(s^{\prime}|s,a)$ is the probability of transitioning to state $s^{\prime}$ from state $s$ using action $a$, $R$ is the reward function such that $R(s,a)$ is the reward for taking action $a$ in state $s$, and $\gamma$ is the discount factor that determines the value of future rewards. A policy $\pi:S\to\Delta(A)$ is a distribution over actions given a state. MDPs can be solved using dynamic programming techniques (e.g., value iteration, policy iteration) which require complete knowledge of the MDP’s dynamics [].

Reinforcement Learning.
Reinforcement Learning (RL) involves training an agent to make a sequence of decisions by interacting with the MDP, which represents the environment. This machine learning technique is used to solve MDPs when full knowledge is not available. The agent’s goal is to find a policy $\pi^{*}$ maximising the expected discounted return $E\left[\sum_{t=0}^{\infty}\gamma^{t}R_{a_{t}}(s_{t},s_{t+1})\right]$ []. The value function $V_{\pi}(s)$ informs the agent how valuable a given state is when following the current policy $\pi$. Common RL algorithms include Q-learning [], and SARSA []. Deep reinforcement learning (DRL) extends traditional algorithms by utilising deep neural networks to approximate the policy $\pi$ or the value function $V$ when the state/action space is high-dimensional and complex. Actor-critic methods [] are a popular class of algorithms both in traditional and deep RL. Distinctly, the policy (actor) and the value function (critic) are modelled as separate components, allowing for simultaneous updates to both functions. Deep Deterministic Policy Gradient (DDPG) [] is an example of an actor-critic method tailored specifically for continuous action spaces.

Contrastive Learning.
Contrastive Learning (CL) [] can be either an unsupervised or semi-supervised machine learning paradigm aiming at distinguishing between similar (positive) and dissimilar (negative) pairs of data. At its core lies a contrastive loss function, which encourages the model to put similar pairs closer together in the embedding space while separating dissimilar pairs. Given a pair of inputs $x_{i}$, $x_{j}$, the contrastive loss function is defined as:

$$ $L(x_{i},x_{j},y,\theta)=y\cdot\lVert h_{\theta}(x_{i})-h_{\theta}(x_{j})\lVert^{2}+(1-y)\cdot max(0,m-\lVert h_{\theta}(x_{i})-h_{\theta}(x_{j})\lVert^{2})$ (1) $$

where the binary label $y$ indicates if the pair is similar ($y=1$) or dissimilar ($y=0$) and the margin $m$ regulates the minimum distance between dissimilar pairs. The loss function encourages the model to learn meaningful representations that reflect the inherent similarities and differences between data points, thus facilitating the formation of well-defined clusters in the embedding space.

## 4 ADVICE

Our ADVICE post-shielding technique empowers safe RL exploration in black-box environments without requiring a system model. Figure [1](#S4.F1) shows a high-level overview of ADVICE, including its key shield construction and execution/ adaptation stages, and its incorporation within the standard RL loop. The core of ADVICE comprises a contrastive autoencoder (CA) model that can efficiently distinguish between safe and unsafe features from the feature space $\mathcal{F}:S\times A$ (representing all possible state-action pairs). A feature $f_{t}=(s_{t},a_{t})$, $f_{t}\in\mathcal{F}$, denotes a state-action pair at time step $t$. The CA model leverages a unique loss function where similar and dissimilar features are compared, enabling the systematic identification of meaningful latent feature representations. ADVICE employs these latent representations and specialises an unsupervised nearest neighbours model to the learnt embedding space, thus enabling the classification of new features. Formally, the ADVICE shield is defined by the function

$$ $\phi:F\times\mathbb{Z}_{+}\rightarrow A$ (2) $$

such that at time step $t$ during the execution of an episode, ADVICE evaluates the agent’s desired action $\phi(f_{t},K)$ and allows the action $a_{t}$ to be taken, or enforces the execution of another safe action $a^{\prime}_{t}$ instead. The ADVICE-specific parameter $K$ enables controlling the risk aversion levels of the shield. In particular, ADVICE considers the agent’s performance and automatically adapts the $K$ value, thus supporting the dynamic calibration of ADVICE’s cautiousness level during learning. Next, we introduce the details of ADVICE and its integration within the RL loop as a post-shield.

Figure: Figure 1: A high-level overview of the ADVICE construction, execution, and adaptation.
Refer to caption: /html/2405.18180/assets/x1.png

### 4.1 ADVICE Shield Construction

As a black-box post-shielding technique, ADVICE does not rely on any prior knowledge about the RL agent or its environment. Instead, the ADVICE shield construction stage is founded on collecting a feature set $F_{E}$ during an initial unshielded interaction period of $E$ episodes where the RL agent is allowed to interact with and collect experience from its environment. A feature $f_{t}\in F_{E}$ is classified as either safe $\mathcal{S}\subset F_{E}$, unsafe $\mathcal{U}\subset F_{E}$ or inconclusive $\mathcal{I}\subset F_{E}$ based on the following function:

$$ $g(f_{t})=\begin{cases}\mathcal{S}&\text{if }t=0\text{ or }s_{t+1}=\text{goal (accepting state)},\\ \mathcal{U}&\text{if }s_{t+1}=\text{terminal (failure state)}.\\ \mathcal{I}&\text{ otherwise}\end{cases}$ (3) $$

hence $\mathcal{S}\cup\mathcal{U}\cup\mathcal{I}=F_{E}$ and $\mathcal{S}\cap\mathcal{U}\cap\mathcal{I}=\emptyset$. The collected features set $F_{E}$ from the initial interaction period $E$ are organised into these categories to facilitate the contrastive learning process: (i) pairs of features that are similar (e.g., two safe/unsafe features); and (ii) feature pairs that are dissimilar (a safe and an unsafe feature). This categorisation is vital to allow the model to discern between safe and unsafe features effectively and focus on finding meaningful representations in a lower-dimensional latent space that reflect the similarities and differences. The combine function $\mathcal{C}$ consumes the set of features in sets $\mathcal{S}$ and $\mathcal{U}$
and produces all pairwise combinations from $\mathcal{S}$ and $\mathcal{U}$:

$$ $\displaystyle\mathcal{C}(\mathcal{S},\mathcal{U})=$ $\displaystyle\left\{(f_{i},f_{j},1)\mid i,j\in S,i\neq j\right\}\cup$ (4) $\displaystyle\left\{(f_{i},f_{j},1)\mid i,j\in U,i\neq j\right\}\cup$ $\displaystyle\left\{(f_{i},f_{j},0)\mid i\in S,j\in U\right\}$ $$

where $(f_{i},f_{j},1)$ indicates a pair of similar features, and $(f_{i},f_{j},0)$ shows a pair of dissimilar features.

The training of the CA model involves using the pairs of collected features and optimising two loss functions simultaneously: the mean squared error (MSE) and the contrastive loss function (CL) presented in Equation ([1](#S3.E1)). The MSE loss measures how accurately the model can reconstruct the features input after encoding and decoding. In contrast, the CL is designed to refine the model’s ability to cluster similar features together, whilst separating dissimilar ones within the latent space based on the Euclidean distance between them. This distance is minimised for similar pairs and maximised for dissimilar pairs, encouraging high cohesion and high separation between similar and dissimilar features, respectively.
Thus, the CA learns to encode and reconstruct the salient features of the given RL problem, comprising the state of the environment and chosen action, in a lower-dimensional latent space as accurately as possible.

Once trained, the CA model is adept at finding nuanced distinctions between safe and unsafe features and accurately placing unseen features within the appropriate partitions in the latent space. The shield construction stage of ADVICE concludes with embedding an unsupervised nearest neighbours (KNN) model in the latent space that classifies new encoded features as safe/unsafe. A visual representation of the latent data encoding, illustrating the separation between in the latent space, is shown in Figure [2](#S5.F2).

### 4.2 ADVICE Execution and Adaptation

Figure: Algorithm 1 ADVICE Execution and Adaptation

Upon the completion of the shield construction phase, ADVICE can be used as a post-shield within the RL loop.
The constructed shield encourages safe environment exploration, guarding the agent throughout its interaction with the environment by ensuring the execution of safe actions.

The ADVICE execution and adaptation stage, shown in Algorithm [1](#alg1) and Figure [1](#S4.F1) (bottom), involves a continuous cycle of action evaluation and decision-making until the maximum number of training episodes $E_{max}$ is reached (line [2](#alg1.l2)).
During an episode’s execution (line [4](#alg1.l4)), the RL agent, having observed the current state $s_{t}$, selects an action $a_{t}$ based on its current policy $\pi$ (line [5](#alg1.l5)).
Without ADVICE, the agent would proceed with this action regardless of its potential safety implications.
Instead, ADVICE can now be used to evaluate the selected action before it is realised by the agent, thus ensuring safe environment exploration.
More specifically, to establish if the action $a_{t}$ is safe, ADVICE extracts the latent representation $\hat{f}$ of feature $f_{t}$, collects the nearest $K^{max}$ latent data points to $\hat{f}$ and checks whether the cardinality of the safe data points exceeds the safety threshold $K$, in which case the action $a_{t}$ is considered safe; otherwise, $a_{t}$ is considered unsafe (lines [15](#alg1.l15)–[19](#alg1.l19)).

The safety threshold $K\in[0,K^{max}]$ determines how many neighbours need to be labelled safe so that the encoded feature can be deemed safe.
The closer $K$ is to $K^{max}$ entails that ADVICE will be more cautious in terms of safety, while a $K$ value closer to $\lceil K/2\rceil$ means that the ADVICE shield is more relaxed and favours exploration.
A value of $K<\lceil K/2\rceil$ should not be considered as this would allow the RL agent to execute actions that are more likely to be unsafe than safe.

If action $a_{t}$ is deemed unsafe (line [6](#alg1.l6)), ADVICE intervenes and generates a set of candidate actions $A_{f_{t}}$ by quantising the continuous space of each action dimension $A^{d}$, where $d\in D$ is the dimension, and extracting the Cartesian product across the $D$ action dimensions (line [7](#alg1.l7)).
The set of candidate actions $A_{f_{t}}$ is filtered, keeping only those actions that are considered safe $A^{\prime}_{f_{t}}\subseteq A_{f_{t}}$ (line [8](#alg1.l8)).
If the $A^{\prime}_{f_{t}}$ set is not empty, these filtered candidate actions are evaluated by the RL agent’s value function $Q_{\pi}$ for their expected reward (line [10](#alg1.l10)).
The action with the highest expected reward is selected, entailing that the action aligns with both the safety considerations and the agent’s performance objectives.
Note that if no safe alternative action is identified (i.e., $A_{f_{t}}^{\prime}=\emptyset$), ADVICE resorts to providing the originally selected action $a_{t}$; since no alternative action is predicted to be safe, $a_{t}$ is expected to achieve the highest expected reward.

At the end of each episode, ADVICE calibrates its cautiousness level by considering the recent safety performance of the RL agent (line [13](#alg1.l13)).
This unique ADVICE characteristic enables moving beyond the static definition of the safety threshold $K$ by automatically adjusting its value in response to the frequency of recent safety violations.
Accordingly, ADVICE becomes adaptive and allows the agent to explore more when exhibiting safe behaviour for a period of time while being more cautious, thus interfering more when the RL agent behaves increasingly unsafely.

To assess the agent’s performance over time, ADVICE employs a double sliding window, commonly used in the field of anomaly detection [, ].
By comparing the recent rate of safety violations against a broader historical view, ADVICE can discern whether the current trend deviates from the pattern seen historically.
This analysis informs ADVICE whether to strengthen or relax the cautiousness level.
The adaptation function is defined as follows:

$$ $K=\begin{cases}min(K\!+\!1,K^{max}),&\text{if }MA_{h_{r}}>MA_{h_{d}}+\sigma_{h_{d}}\\ max(K\!-\!1,\lceil K^{max}/2\rceil),&\text{if }MA_{h_{r}}<MA_{h_{d}}-\sigma_{h_{d}}\\ K,&\text{otherwise}\end{cases}$ (5) $$

where $MA_{h_{d}}$ and $\sigma_{h_{d}}$ are the moving average and standard deviation, respectively, of safety violations $Z_{i}$. This uses the distant history window $h_{d}$, and $MA_{h_{r}}$ is the moving average of $Z_{i}$ using the recent history window $h_{r}$, i.e., $h_{r}<h_{d}$.
Given $h\in\{h_{r},h_{r}\}$, the calculation of the moving average $MA_{h}$ and standard deviation $\sigma_{h}$, considering the history of safety violations $\mathcal{Z}_{1},\dots,\mathcal{Z}_{t}$, is defined by:

$$ $MA_{h}=\frac{1}{h}\sum_{i=t-h}^{t}(\mathcal{Z}_{i}-\mathcal{Z}_{i-1})$ (6) $$

$$ $\sigma_{h}=\sqrt{\frac{1}{h}\sum_{i=t-h}^{t}\left((\mathcal{Z}_{i}-\mathcal{Z}_{i-1})-MA_{h}\right)^{2}}$ (7) $$

If the $MA_{h_{r}}$ rate exceeds the sum of $MA_{h_{d}}+\sigma_{h_{d}}$, then it signifies that the agent crashes more often than expected.
Consequently, $K$ is automatically incremented to adopt a more cautious stance.
Inversely, if the $MA_{h_{r}}$ rate is below $MA_{h_{d}}-\sigma_{h_{d}}$, this suggests that the agent acts more conservatively than expected.
As a result, $K$ is automatically reduced to allow the agent to explore more freely.
The safety threshold $K$ remains the same for any other occasion.
ADVICE deliberately only considers one standard deviation as to two or more standard deviations would make the adaptation slow to respond to emerging safety risks. Our sensitivty analysis on these two parameters (Section [5.3](#S5.SS3)),
demonstrates the described behaviours.

## 5 Evaluation

We evaluate ADVICE (^1^11The ADVICE implementation is available at: [https://github.com/team-daniel/ADVICE](https://github.com/team-daniel/ADVICE)) against a set of established methods using an array of tasks within the Safety Gymnasium [] test-suite involving a robot with Lidar sensors that must complete a given task by navigating through environments with obstacles.
This benchmark suite is designed to assess performance and safety in high-dimensional, complex scenarios that incorporate a variety of safety constraints.
Accordingly, the Safety Gymnasium is an ideal benchmark to assess the effectiveness of ADVICE.
Each task is designed to terminate the episode immediately upon a collision, signifying catastrophic damage to a physical robot. The selected tasks below encompass a range of complexities:

- •
Semi-random Goal:
A standard goal environment with six obstacles and one goal. The obstacles have a static spawn, while the agent and the goal have randomised positions every episode.
- •
Randomised Goal:
A similar goal environment, but obstacles have random locations per episode.
- •
Randomised Circle: The agent has to circle in a given zone in this environment. The aim is to maximise speed and distance from zone’s centre while avoiding the three randomised obstacles.

Further details of these three environments and the corresponding tasks are provided in Appendix [A](#A1).
Figure [6](#A1.F6) shows a snapshot of the tasks within the Safety Gymnasium.
To evaluate the performance of ADVICE, we conduct comparisons against the following state-of-the-art algorithms:

- •
DDPG: A deep deterministic policy gradient (DDPG) [] agent being the foundational baseline.
- •
DDPG-Lag: A DDPG agent with a Lagrangian multiplier ($\lambda=0.1$, $a=0.01$) which heavily penalises constant safety violations.
- •
Tabular Shield: A DDPG agent with a discretised (1 decimal place) table of terminal state-action pairs, which prevents the agent from executing inadequate actions again [].

Each method employs the same DDPG configuration, ensuring a fair and accurate comparison. Detailed information per algorithm structure and hyperparameter settings for all experiments is available in Appendix [E](#A5).
The results are averaged across three independent runs and include mean scores and confidence intervals (standard error of the mean), showing a performance overview.
Unless explicitly stated, the default ADVICE deployment per run is $E=1000$ with $K=4$ and $K^{max}=5$.

### 5.1 Performance Results

Figure: (a) Semi-random goal environment results
Refer to caption: /html/2405.18180/assets/Images/subplot_legend_large.png

For our first set of experiments, we evaluate the overall performance of the examined methods (DDPG, DDPG-Lag, Tabular shield, ADVICE) in all three tasks.
Figure [2](#S5.F2) shows the average episodic reward (return), cumulative safety violations (failed episodes), and cumulative goal reaches (successful episodes), alongside an example latent space visualisation from a single ADVICE execution.
The obtained results provide evidence that ADVICE, albeit designed to prioritise safety, manages to maintain a competitive performance (rewards) in all tasks.
Both DDPG-Lag and ADVICE, indicate an inherent trade-off between reward maximisation and safety prioritisation.

Since the primary goal of a safety shield is to encourage exploration and learning in a safe way, it is evident that ADVICE, across all tasks, can significantly reduce safety violations, outperforming by a notable margin all other methods.
DDPG-Lag also manages to reduce safety violations, but not to the magnitude of ADVICE, highlighting the effectiveness of ADVICE’s shield.
We visualise the learnt latent space from an example ADVICE run (Figure [2](#S5.F2) right)
illustrating the power of contrastive learning and validating its effectiveness at learning latent representations that empower the distinction between safe and unsafe features.
Considering that ADVICE constructs the shield for the first $E\!=\!1000$ episodes and then starts its execution for safe exploration, while DDPG-Lag aims at reducing safety violations from episode $E\!=\!1$, the results are even more affirmative.

Despite its conservative prioritisation to reward, ADVICE still reaches the goal (successful episode execution) with a similar frequency to all other methods.
In fact, ADVICE completes the task without compromising safety, and the difference in accumulated reward is due to the reward function design and the reduced time the agent has to complete the task using a cautious path.
Through additional experiments with increased number of maximum steps per episode (Figure [12](#A3.F12) - Appendix C), we have established that ADVICE’s average episodic return is very similar to the baseline methods.
These core results corroborate that ADVICE’s contrastive approach to shielding an RL agent from unsafe actions is effective in comparison to other methods.
ADVICE significantly reduces safety violations without detracting much from the agent’s overall performance.

Figure: (a) DDPG agent
Refer to caption: /html/2405.18180/assets/x2.png

Figure [3](#S5.F3) visualises example trajectories of the evaluated methods on the semi-random goal environment, further validating our core findings. The DDPG and Tabular shield agents focus primarily on maximising expected return, and, thus, navigate through the centre of the wall of objects to reach the goal. Alongside action noise, this is risky as the agent may accidentally collide with an object, as shown in Figure [2(a)](#S5.F2.sf1). The DDPG-Lag agent tends to learn similar behaviours, except it leaves a larger gap between itself and the objects, considering the possibility of action noise. Finally, ADVICE adopts a cautious approach, learning to navigate the long way around the wall of objects. Learning this path ensures that the agent is less likely to collide with an obstacle. However, this incurs a longer trajectory and a slightly reduced reward due to moving further away from the goal initially.

Figure: Figure 4: Average episodic reward, cumulative safety violations, and cumulative goal reaches on the semi-randomised goal environment where ADVICE and Tabular Shield have a pre-trained shield.
Refer to caption: /html/2405.18180/assets/Images/Appendix/appendix_transfer_learning.png

### 5.2 Transfer Learning

Figure [4](#S5.F4) shows the transfer learning capabilities of ADVICE using the semi-random environment.
ADVICE and Tabular shield use a pre-trained shield from the randomised environment and are then deployed on a new environment with different goal and agent positions.
Using the pre-trained shield, ADVICE can immediately reduce safety violations; in contrast, all other methods struggle.
Over the 1000 episodes, ADVICE has reduced violations by over 50%, highlighting ADVICE’s ability to generalise the constructed shield across similar environments despite data from the randomised environment being subtly different to features in the semi-randomised environment.
This significant violations reduction in an environment not trained in shows ADVICE’s robustness and flexibility
that other methods do not provide.
The results signify that ADVICE is useful in non-stationary domains or in those where conditions may vary.
Whilst promoting safety, ADVICE accumulates a comparable reward to the other methods achieving the same frequency of goal reaches.
These results provide strong empirical evidence that
ADVICE’s shield can comfortably support transfer learning tasks.

### 5.3 ADVICE Adaptation

To evaluate ADVICE’s adaptation capabilities, we conducted a sensitivity analysis focusing on the impact of varying the distant history window $h_{d}$ and the recent history window $h_{r}$.
For each $h_{d}$ and $h_{r}$ combination, Table [1](#S5.T1) shows the number of consecutive episodes $K$ was fixed at a specific value, the frequency of $K$ adjustments, and the impact on mean safety violations.

Our findings reveal an impact of $h_{r}$ on the frequency of $K$ adjustments.
A smaller $h_{r}$ leads to more frequent $K$ increases, allowing the system to quickly respond to immediate safety violations. Conversely, a larger $h_{r}$ tends to stabilise $K$ by filtering out anomalies and adjusting only in response to sustained trends of increased violations. Similarly, $h_{d}$ influences the decrease of $K$; a smaller $h_{d}$ facilitates rapid decreases in $K$ following a reduction in safety violations, whereas a larger $h_{d}$ results in less frequent reductions, promoting stability in ADVICE’s behaviour.
The interaction between $h_{d}$ and $h_{r}$ minimally affects the overall rate of safety violations, suggesting that while these parameters impact the adaptiveness and stability of $K$, they do not directly correlate with safety violations. These insights highlight the role of $h_{d}$ and $h_{r}$ primarily as tuning parameters to balance responsiveness against stability in ADVICE.

**Table 1: Mean sensitivity analysis of $h_{d}$ and $h_{r}$ on the randomised goal environment for $E^{max}\!=\!1000$**
|  | $h_{d}$ |  |  |
| --- | --- | --- | --- |
| $h_{r}$ | Metrics | 10 | 25 |
| 2 | Consecutive Episodes ($K=3$) | $24.49\pm 9.59$ | $31.09\pm 4.53$ |
| Consecutive Episodes ($K=4$) | $1.00\pm 0.00$ | $7.89\pm 0.77$ |  |
| Consecutive Episodes ($K=5$) | $32.97\pm 7.95$ | $151.60\pm 3.09$ |  |
| Changes of $K$ | $116.33\pm 9.87$ | $27.67\pm 4.62$ |  |
| 2 | Safety Violations | $50.42\pm 9.94$ | $57.88\pm 7.64$ |
|  | Consecutive Episodes ($K=3$) | $166.81\pm 5.71$ | $117.35\pm 8.07$ |
|  | Consecutive Episodes ($K=4$) | $21.10\pm 6.86$ | $28.76\pm 9.26$ |
|  | Consecutive Episodes ($K=5$) | $13.49\pm 5.40$ | $149.68\pm 8.71$ |
|  | Changes of $K$ | $28.67\pm 1.41$ | $19.33\pm 2.83$ |
| 3 | Safety Violations | $51.69\pm 6.93$ | $54.45\pm 5.10$ |
|  | Consecutive Episodes ($K=3$) | $954.49\pm 9.59$ | $169.07\pm 9.02$ |
|  | Consecutive Episodes ($K=4$) | $22.60\pm 6.71$ | $22.10\pm 7.26$ |
|  | Consecutive Episodes ($K=5$) | $0.00\pm 0.00$ | $181.35\pm 4.92$ |
|  | Changes of $K$ | $2.33\pm 0.47$ | $17.77\pm 2.72$ |
| 4 | Safety Violations | $55.48\pm 5.64$ | $50.17\pm 5.47$ |

## 6 Limitations

Although ADVICE maintains a lower safety violation rate compared to other methods, the following aspects deserve further investigation.
Firstly, ADVICE suffers from cold-start needing to gather sufficient features to train the contrastive autoencoder.
Although this delays its effectiveness, it still significantly reduces violations compared to methods that start learning from timestep $t_{0}$.
Transfer learning, i.e., exploiting shields from other agents, mitigates this issue (Figure [4](#S5.F4)).
ADVICE increases computational overheads due to continuous inference at each timestep, possibly limiting its applicability in resource-limited settings.
Inferring the safety of actions at alternate timesteps could reduce overheads, assuming minimal changes between consecutive states, but may also delay detecting emerging unsafe conditions.
Thirdly, in environments with temporally dynamic obstacles, ADVICE could be less effective in reducing violations as it only captures a single environmental snapshot per timestep, lacking the temporal context.
Integrating LSTM or features from previous states and actions could help, although this extension could further elevate computational requirements.
Finally, additional experiments could enable establishing the applicability, feasibility and scalability of ADVICE in domains with characteristics different from those used in our evaluation.

## 7 Conclusion and Future Work

We introduced ADVICE, a post-shielding technique for the safe exploration of RL agents operating in complex black-box environments.
ADVICE does not need *any* prior knowledge and uses a contrastive autoencoder to distinguish between safe and unsafe features efficiently.
Our evaluation shows that ADVICE significantly reduces safety violations while maintaining competitive performance against state-of-the-art methods.
Despite its effectiveness, ADVICE has areas for improvement, including mitigating the cold-start issue, reducing high computational demands, and addressing challenges with dynamic obstacles.
Future work could explore meta-learning techniques [] to address the cold-start problem, enabling faster activation of ADVICE without sacrificing performance.
Quantisation or pruning could reduce computational demands, enhancing applicability in computationally constrained domains.
We envisage that ADVICE serves as a foundational step in using neural network-based shielding for safe RL exploration in complex, black-box environments without *any* prior knowledge.

## Appendix A Task Details

In the Safety Gymnasium [] (^2^22More details here: https://github.com/PKU-Alignment/safety-gymnasium) test-suite, a robot with Lidar sensors has to navigate through environments with obstacles to complete a given task. The test suite comes with a multitude of robots (e.g. Point, Ant, Car) and a set of tasks (e.g. Goal, Circle, Button) that can be evaluated. In our experiments, found in Section [5](#S5), we chose to use:

- •
Car robot: This robot has two wheels on the rear that the agent can control with one free-rolling front wheel. Steering and movement require nuanced coordination. The action space for the car is $\left[-1,1\right]^{2}$, and the agent is shown in Figure [5](#A1.F5).
- •
Semi-random Goal: A standard goal environment, where the agent aims to reach the goal at the end of the episode whilst navigating through six obstacles. The six obstacles have a static spawn, the agent and the goal have randomised positions every episode. We placed the six obstacles to form a large wall, where the agent can fit through to reach to goal but with increased risk of crashing. In this instance, we want to determine if ADVICE and other safe RL exploration methods will learn to avoid the wall or risk going through it. Deployment trajectories in Figure [3](#S5.F3), show the learnt trajectories of ADVICE and other methods.
- •
Randomised Goal: This environment is similar to the semi-random goal environment, with the additional complexity of the obstacles also having random spawns. This extra randomised aspect adds increased difficulty as the agent and safety mechanisms cannot memorise the positions of the obstacles to avoid.
- •
Randomised Circle: The agent has to circle in a given zone in this environment. The aim is to maximise speed and distance from the centre of the zone whilst navigating through three randomised obstacles. This scenario element further increases the task difficulty as now the obstacles to avoid are directly within the area where the agent can maximise its reward.

In all environments for all the tasks above, the agent uses psuedo Lidar to perceive objects in the environment. Each type of object (e.g. goal, obstacles) in the environment has its own separate Lidar observation, where a Lidar vector has 16 bins. All vectors are flattened into one observational vector and then given to the agent as the current state. For example, in the semi-random goal environment, there is a goal and a set of obstacles. Here the observational space is $\left[0,1\right]^{32}$. All lidar vectors have a max distance of $3$ meters. In both goal environments, an episode has a maximum timestep of $1000$. In the circle environment, the maximum timestep is $500$. A goal reach in both goal environments is defined as reaching the goal before the episode truncates, in the circle environment it is defined as being within the circle when the episode truncates. A crash is defined as the agent colliding with an obstacle, when this occurs, the episode is terminated shortly. Anything else is considered as the episode timing out.

Figure: (a) Front
Refer to caption: /html/2405.18180/assets/Images/Appendix/car_front.jpeg

Each task has a separate reward function for the agent to maximise. Whenever the agent comes into contact with an obstacle, a constraint cost of $-1$ is given. The reward functions for each task are:

- •
Semi-random & random Goal: $R_{t}=(D_{last}-D_{now})\beta$, where $D_{last}$ is the distance between the agent and the goal in timestep $t_{-1}$, $D_{now}$ is the distance between the agent and the goal in timestep $t$, and $\beta$ is the discount factor. Simply, the agent moving towards the goal, in terms of Euclidean distance, gains a positive reward. The agent moving away from the goal gains a negative reward. Reaching the goal gains a static reward of $+1$.
- •
Randomised Circle: $R_{t}=\frac{1}{1+|r_{agent}-r_{circle}|}*\frac{(-uy+vx)}{r_{agent}}$ where $(u,v)$ is the x-y velocity coordinates of the agent, $(x,y)$ are the x-y coordinates, $r_{agent}$ is the Euclidean distance of the agent from the origin of the circle, and $r_{circle}$ is the radius of the circle. Simply, the agent is rewarded for moving at speed along the circumference of the circle.

It should be emphasized that the agent or methods used in the evaluation have no prior knowledge of the task/environment/safety concern. Thus, we can define this environment and all tasks within as black box.

Figure: (a) Semi-random Goal
Refer to caption: /html/2405.18180/assets/Images/example_goal_static.png

## Appendix B Full Training Results

In Section [5.1](#S5.SS1), we show the main results for all methods in a set of tasks. For fair comparison, we show results from episode $1000$ and standardise all metrics to zero. Below, in Figure [7](#A2.F7), we show the unstandardised results for the same experiments.

Figure: (a) Full results on semi-random goal environment.
Refer to caption: /html/2405.18180/assets/Images/Appendix/appendix_semirandom.png

Figure: Figure 8: The average shield activations for ADVICE and the Tabular Shield in the semi-random goal, random goal, and circle environments respectively.
Refer to caption: /html/2405.18180/assets/Images/Appendix/appendix_shield_acts.png

In all experiments, the Tabular Shield method performs approximately the same as the standard DDPG agent. To show why this behaviour occurs, we plot the average shield activations for ADVICE and the Tabular Shield in Figure [8](#A2.F8).
From these results, it is evident that the Tabular Shield does not once activate during training across all tasks. This is due to the high dimensionality of the environments evaluated. Even though the features stored are discretised to $1$ decimal place, the agent has to observe the exact same values across all $\approx 32$ dimensions plus the actions for the shield to activate. Our experiments show that this method fails in these types of environments. A trend that can be noticed with ADVICE is when the shield is first activated, the amount of interventions starts relatively high. As training progresses this number reduces, which shows that the agent learns to adapt to the shield’s understanding of safety.

Figure: Figure 9: The rate of safety violations for all methods in the semi-random goal, random goal, and circle environments respectively.
Refer to caption: /html/2405.18180/assets/Images/Appendix/appendix_safety_viols.png

Figure [9](#A2.F9) shows the rate of safety violations during training. This outcome further validates the results and conclusions discussed in Section [5.1](#S5.SS1). We observe that the DDPG and Tabular Shield agents perform similarly. The DDPG-Lag agent manages to reduce the safety violations somewhat towards the end of training, which is particularly evident in the circle environment. Once ADVICE is turned on, it significantly reduces the rate of safety violations in all environments.

## Appendix C Parameter Analysis

In this section, we present an extended analysis of ADVICE to display the robustness and adaptability of the approach. These experiments were chosen to explore the effects of varying $K$ thresholds, and the timing of ADVICE’s activations $E$.

Figure: Figure 10: Average episodic reward, cumulative safety violations, and cumulative goal reaches of various values of $K$ on the randomised goal environment.
Refer to caption: /html/2405.18180/assets/Images/neighbours_legend_large.png

A user can specify the conservativeness of ADVICE using the safety threshold $K$. In Figure [10](#A3.F10), we evaluate how this parameter affects the model’s safety, and performance. The results are clear, increasing $K$ leads to a more conservative behaviour as hypothesised. The reward decreases a small amount as well as the cumulative goal reaches, however, it also results in fewer safety violations. Conversely, decreasing $K$ allows the underlying DDPG agent more freedom. As a result, average reward and goal reaches are increased at the expense of safety violations. These findings display a clear trade-off between return efficiency and safety assurance.

Figure: Figure 11: The rate of safety violations, and rate of goal reaches when ADVICE is activated at different intervals $E$.
Refer to caption: /html/2405.18180/assets/Images/episode_legend_large_v2.png

ADVICE has a cold-start, meaning it requires some period of time before activation to collect data in order to work efficiently. We acknowledge that this can affect the performance of ADVICE  evidently we show the results of various activation points in Figure [11](#A3.F11). To allow for a fair comparison as possible, we show the rate of safety violations for the subsequent $1000$ episodes after activation. Again, we visualise a trade-off. Delaying ADVICE’s activation for longer results in fewer safety violations and increased goal reaches. However, the RL agent observes more cumulative safety violations up to the point of activation. Starting ADVICE earlier decreases the number of safety violations up to activation but gives the autoencoder fewer data points to train off. Evidently, safety violations are not reduced to the same magnitude and goal reaches also decrease. This is to be expected with any neural network-based approach.

Figure: Figure 12: Average episodic reward, cumulative safety violations, and cumulative goal reaches of various methods on the randomised goal environment where the maximum episodic steps are doubled.
Refer to caption: /html/2405.18180/assets/Images/steps_legend_large_v2.png

Based on results in Figure [2(b)](#S5.F2.sf2), we hypothesise that the reduced reward and cumulative goal reaches is a result of ADVICE not having enough time to complete the task. As seen in Figure [3](#S5.F3), ADVICE learns to take a longer route to the objective, so by doubling the maximum step count allowed per episode, we expect to see an increase in cumulative goal reaches, average reward, and no increase to safety violations. Results for this experiment are shown in Figure [12](#A3.F12). As expected, given more time to complete the task, ADVICE now reaches the goal more than when the maximum step counter is the default value. As a result, we observe an increase in average episodic return much closer to baseline methods.

## Appendix D Adaptive ADVICE

To validate that Adaptive ADVICE correctly increases and lowers $K$ during training, we plot an example visualisation window in Figure [13](#A4.F13) showing the rate of safety violations, the upper $(MA_{h_{d}}+\sigma_{h_{d}})$ and lower $(MA_{h_{d}}-\sigma_{h_{d}})$ thresholds, the moving average $(MA_{h_{r}})$, and the value of $K$.

Figure: Figure 13: The rate of safety violations, lower threshold $(MA_{h_{d}}-\sigma_{h_{d}})$, upper threshold $(MA_{h_{d}}+\sigma_{h_{d}})$, recent moving average $(MA_{h_{r}})$, and value of $K$ during an example run where $h_{d}=10$ and $h_{r}=2$.
Refer to caption: /html/2405.18180/assets/Images/Appendix/appendix_adapt_example.png

It can be seen that when the recent moving average $(MA_{h_{r}})$ is above the upper threshold $(MA_{h_{d}}+\sigma_{h_{d}})$, the adaptive module correctly increments $K$. An example of this can be seen at episode $1553$. The agent crashes, and both thresholds adjust but the recent moving average climbs above the upper threshold, increasing $K$ as a result. In subsequent episodes afterwards, the recent moving average falls between both thresholds. Here the adaptive module correctly keeps $K$ at the same value until episode $1564$ where the moving average is equal to the lower threshold. As a result, $K$ is decreased. This example window validates that the adaptive shield works as expected and also provides an insight into how it works during training.

Figure: Figure 14: Rate of safety violations and value of $K$ on an example run (one random seed) that shows the adaptation of ADVICE ($h_{d}=10,h_{r}=2$).
Refer to caption: /html/2405.18180/assets/Images/Appendix/appendix_adapt_example_ten_two.png

## Appendix E Hyperparameter Analysis and Computational Overheads

This section lists the hyperparameters used by all models and ADVICE. Table [2](#A5.T2) summarises all hyperparameters used in Section [5](#S5). We will refer the reader to our source code repository [1](#footnote1) for the remaining details.

Using all model configurations in Table [2](#A5.T2), a single ADVICE run (one random seed) takes 12, 24, and 12 hours of training, respectively, in the semi-random goal, random goal, and random circle environments. For all other methods, a single run takes 2, 3, and 2 hours in the same environments, respectively. All experiments were run on a large computing cluster utilising two Nvidia H100 GPUs, 16 CPUs, and up to 500GB memory.

**Table 2: Summary of hyperparameters in the DDPG algorithm and the ADVICE shield.**
| Parameter | DDPG |  | Parameter | ADVICE Shield |
| --- | --- | --- | --- | --- |
| Network size | (256, 256) |  | Size of network | (512, 2, 512) |
| Optimizer | Adam |  | Optimizer | NAdam |
| Actor learning rate | 2e-3 |  | Learning rate | Reduce on plateau |
| Critic learning rate | 1e-3 |  | Batch size | 32 |
| Size of replay buffer | 2e5 |  | Max epochs | 500 |
| Batch size | 64 |  | No. Neighbours ($K^{max}$) | 5 |
| Gamma | 0.95 |  | No. Safe neighbours ($K$) | 4 |
| Tau | 5e-3 |  | Losses | (MSE, MSE, CL) |
| Ornstein-Uhlenbeck noise | 0.2 |  | Loss weights | (1, 1, 1.25) |
| - | - |  | Unshielded Episodes ($E$) | 1000 |

Hyperparameters for the DDPG algorithm started with author recommendations []. They were manually tuned afterwards to achieve a high performance on individual environments before tests were carried out, meaning the RL algorithm for all approaches was of high performance and fair comparison. Hyperparameters for ADVICE were manually tuned for performance in CL loss and MSE loss. Some hyperparameter analysis was conducted in Section [C](#A3) to justify certain choices. Parameters for the DDPG-Lag method started with recommendations [] and were tuned for performance in our experiments.

## NeurIPS Paper Checklist

- 1.
Claims
- Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
- Answer: [Yes]
- Justification: The claims made within our abstract and introduction are as follows: ADVICE can significantly reduce safety violations during training, it achieves a competitive reward, and to the best of our knowledge, ADVICE is the first work that investigates shielding for safe RL exploration in black-box environments with high-dimensional state/action spaces and proposes an end-to-end methodology for shield synthesis without prior knowledge. Section [5](#S5) demonstrates that ADVICE successfully reduces safety violations during training and achieves a competitive reward in comparison to state-of-the-art methods (DDPG [], DDPG-Lag, Tabular Shield []). Section [2](#S2) summarises prior work that further corroborates our last claim.
- Guidelines:
•
The answer NA means that the abstract and introduction do not include the claims made in the paper.
•
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
•
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
•
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
- 2.
Limitations
- Question: Does the paper discuss the limitations of the work performed by the authors?
- Answer: [Yes]
- Justification: Section [6](#S6) details the limitations of our ADVICE, including the cold-start challenge, increased computational overheads, and reduced applicability in environments with temporally dynamic obstacles.
Each limitation is discussed and supported with candidate mitigation techniques.
- Guidelines:
•
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
•
The authors are encouraged to create a separate "Limitations" section in their paper.
•
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
•
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
•
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
•
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
•
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
•
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
- 3.
Theory Assumptions and Proofs
- Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
- Answer: [N/A]
- Justification: This research focuses on the practical implementation of a post-shielding technique for safe RL agent exploration in black-box environments without *any* prior knowledge. We do not present formal results that require proofs but empirically compare ADVICE using state-of-the-art RL methods (DDPG [], DDPG-Lag, Tabular shield []).
- Guidelines:
•
The answer NA means that the paper does not include theoretical results.
•
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
•
All assumptions should be clearly stated or referenced in the statement of any theorems.
•
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
•
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
•
Theorems and Lemmas that the proof relies upon should be properly referenced.
- 4.
Experimental Result Reproducibility
- Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
- Answer: [Yes]
- Justification: We provide the full experimental setup and algorithm configuration in Section [E](#A5).
We also discuss in detail the environments used in Section [A](#A1). Finally, the open-source ADVICE implementation is provided at [1](#footnote1).
- Guidelines:
•
The answer NA means that the paper does not include experiments.
•
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
•
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
•
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
•
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a)
If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b)
If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c)
If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
- 5.
Open access to data and code
- Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
- Answer: [Yes]
- Justification: We provide access to all of our code, which can be found at [1](#footnote1).
The codebase is in a GitHub repository with detailed instructions to download the Safety Gymnasium simulator locally, download ADVICE’s dependencies, and run ADVICE.
Hyperparameters to replicate the experiments reported in the paper can be found in Section [E](#A5) and made available within ADVICE’s codebase.
- Guidelines:
•
The answer NA means that paper does not include experiments requiring code.
•
Please see the NeurIPS code and data submission guidelines ([https://nips.cc/public/guides/CodeSubmissionPolicy](https://nips.cc/public/guides/CodeSubmissionPolicy)) for more details.
•
While we encourage the release of code and data, we understand that this might not be possible, so “No” is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
•
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ([https://nips.cc/public/guides/CodeSubmissionPolicy](https://nips.cc/public/guides/CodeSubmissionPolicy)) for more details.
•
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
•
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
•
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
•
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
- 6.
Experimental Setting/Details
- Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
- Answer: [Yes]
- Justification: Full training details can be found in Appendix [E](#A5) supported by suitable justification for the choices made. Experimental results where performance parameter analysis has been carried out are also provided in Appendix [C](#A3).
- Guidelines:
•
The answer NA means that the paper does not include experiments.
•
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
•
The full details can be provided either with the code, in appendix, or as supplemental material.
- 7.
Experiment Statistical Significance
- Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
- Answer: [Yes]
- Justification: In all experiments, the mean and confidence intervals (in the form of the standard error of the mean) are reported. In Table [1](#S5.T1), we report the mean and standard deviation.
- Guidelines:
•
The answer NA means that the paper does not include experiments.
•
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
•
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
•
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
•
The assumptions made should be given (e.g., Normally distributed errors).
•
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
•
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
•
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
•
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
- 8.
Experiments Compute Resources
- Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
- Answer: [Yes]
- Justification: We provide details regarding the computational setup used for the empirical evaluation in Appendix [E](#A5). We also discuss the computational overheads as a potential limitation in Section [6](#S6).
- Guidelines:
•
The answer NA means that the paper does not include experiments.
•
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
•
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
•
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).
- 9.
Code Of Ethics
- Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics [https://neurips.cc/public/EthicsGuidelines](https://neurips.cc/public/EthicsGuidelines)?
- Answer: [Yes]
- Justification:
Our research does not use human participants, does not create datasets or uses external datasets. Further, we do not foresee our research having a negative societal impact. Our codebase has appropriate licenses and is available as open-source.
- Guidelines:
•
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
•
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
•
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
- 10.
Broader Impacts
- Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
- Answer: [No]
- Justification: We do not discuss potential positive or negative societal impacts in this paper. The impacts that this research could have are common in the majority of machine learning research, and, therefore, it is not a unique issue to ADVICE.
- Guidelines:
•
The answer NA means that there is no societal impact of the work performed.
•
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
•
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
•
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
•
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
•
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
- 11.
Safeguards
- Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
- Answer: [N/A]
- Justification: This paper poses no such risks.
- Guidelines:
•
The answer NA means that the paper poses no such risks.
•
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
•
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
•
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
- 12.
Licenses for existing assets
- Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
- Answer: [Yes]
- Justification: Original creators of approaches and test suites used within this paper are properly cited, and URLs are given.
- Guidelines:
•
The answer NA means that the paper does not use existing assets.
•
The authors should cite the original paper that produced the code package or dataset.
•
The authors should state which version of the asset is used and, if possible, include a URL.
•
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
•
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
•
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, [paperswithcode.com/datasets](paperswithcode.com/datasets) has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
•
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
•
If this information is not available online, the authors are encouraged to reach out to the asset’s creators.
- 13.
New Assets
- Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
- Answer: [Yes]
- Justification:
The publicly-available ADVICE GitHub repository is available at  [1](#footnote1).
The code is augmented with appropriate comments and details for its execution.
Also, information regarding ADVICE’s training can be found in Appendix [E](#A5).
- Guidelines:
•
The answer NA means that the paper does not release new assets.
•
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
•
The paper should discuss whether and how consent was obtained from people whose asset is used.
•
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
- 14.
Crowdsourcing and Research with Human Subjects
- Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
- Answer: [N/A]
- Justification: Our research does not involve any crowdsourcing nor human subjects.
- Guidelines:
•
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
•
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
•
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
- 15.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
- Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
- Answer: [N/A]
- Justification: Our research does not involve any crowdsourcing nor human subjects.
- Guidelines:
•
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
•
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
•
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
•
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - If the problem had a single initial state, would this be an issue for this approach? How does the diversity of initial states influence the performance of the ADVICE algorithm?
soundness: 2
strengths: - The paper is well written. The description of the method is precise.
- The proposed method is original. It is the first method to use contrastive learning in a safe RL setting.
- The empirical evaluation indicates the approach can potentially increase the safety of RL algorithms.
summary: This paper develops a new shield based on contrastive learning to increase the safety of RL agents. This approach learns a latent representation that separates safe and unsafe state-action pairs. After learning this latent representation, it uses a KNN approach to classify, which requires a threshold K to indicate how many neighbors to consider a state-action pair unsafe. The paper shows how this parameter can be adapted online based on the safety of the RL agent. Finally, the paper provides empirical evidence that the proposed method increases the safety of the RL agent.
weaknesses: The problem formulation is incomplete. The paper does not define the safety properties expected from the RL agent.
- Lack of theoretical results. This paper provides only empirical results to support its claims.
- The results are presented in a convoluted way. In particular, the results disregard the safety violations of the agent in the first 1000 episodes. The reason for presenting the results in this way is unclear.
- The presentation of the DDPG-Lag as a constrained RL algorithm is imprecise, as it uses a fixed weight for the costs, which works as simple reward engineering. In general, with a Lagrangian relaxation, this weight should be adjusted online to ensure the accumulated cost stays below a predefined threshold [1].
- The evaluation in CMDPs is inconsistent. These approaches solve different problems where a predefined accumulated cost is allowed.
- Weak baseline. From the results in Figure 10, it is clear that Tabular Shield does not recognize any unsafe state-action pairs, making it an unsuitable baseline. This is not surprising considering how the state-action space is discretized. Perhaps it is necessary to finetune the discretization of this baseline. Alternatively, it would be more suitable to consider stronger baselines, such as the accumulating safety rules [2]

**references**
- [1] Ray, A., Achiam, J., and Amodei, D. (2019). *Benchmarking safe exploration in deep reinforcement learning*. <https://github.com/openai/safety-gym>
- [2] Shperberg, S. S., Liu, B., Allievi, A., and Stone, P. (2022). A rule-based shield: Accumulating safety rules from catastrophic action effects. *CoLLAs*, 231–242. <https://proceedings.mlr.press/v199/shperberg22a.html>

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: - Can ADVICE be integrated with other RL algorithms except for DDPG?
- What are the potential limitations of using a contrastive autoencoder in the context of safe RL, and how might these be addressed?
- What are the implications of ADVICE's cold-start issue on its applicability in real-world scenarios where immediate safety is critical?
soundness: 3
strengths: - ADVICE introduces a new way to handle safety in RL using a contrastive autoencoder for distinguishing safe and unsafe actions.
- Despite prioritizing safety, ADVICE maintains competitive performance in terms of rewards compared to other methods.
-  ADVICE does not require prior knowledge about the environment, making it suitable for black-box scenarios.
summary: The paper presents ADVICE (Adaptive Shielding with a Contrastive Autoencoder), a novel post-shielding technique for safe reinforcement learning (RL) in complex black-box environments. ADVICE distinguishes between safe and unsafe state-action pairs during training using a contrastive autoencoder, protecting the RL agent from executing hazardous actions. The method includes an adaptation component based on the agent's recent performance, encouraging exploration when appropriate. Extensive experiments against state-of-the-art safe RL exploration techniques demonstrate that ADVICE significantly reduces safety violations during training while maintaining competitive outcome rewards.
weaknesses: - ADVICE requires an initial period to gather data before it can be fully effective, which could be a disadvantage in some scenarios.
- The paper suggests that ADVICE might struggle with dynamic environments and could benefit from incorporating temporal context, which would add additional computational load.
- The performance of ADVICE is sensitive to hyperparameters like the safety threshold K, which might require careful tuning.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: - What are the data of visualization in last column in fig.2? Are they training data for neighbor model?
- Does the "constrained randomised goal" correspond to "randomised CMDP" in fig.7(d)?
- Why do not you use the standard tasks provided by safety-gymnasium (e.g., PointGoal, CarButton, ...) instead of customizing the tasks? The original tasks should be harder because all the layouts (agent, goal and obstacles) are randomized in each episode and they can better test the ability of safety exploration.
- In the experiment, why are the performances of DDPG and DDPG-lag very similar in terms of reward or safety violation? DDPG does not take safety into the consideration and only aims to maximize the reward. The lagrangian in DDPG-lag will thus be almost useless if it performs similarly to DDPG.
soundness: 2
strengths: - This paper proposes a new shield-based method for safe exploration, which is an important problem for application of RL.
- Overall, the paper is clearly written (e.g., fig. 1) and easy to follow. 
- The idea of classifying the safety of state-action in latent space is novel.
summary: This paper proposes a post-shielding method ADVICE to reduce the constraint violation during the safe RL training. It first trains a neighbor model to classify the safety of state-action pair by contrastive learning in embedding space. Then it leverages this model to identify the safety of new state-action by the statistics of its K nearest neighbors and corrects the unsafe action to a safe one from a selected safe set. The authors conduct experiments on several safety gymnasium environments, which show that the proposed method can reduce the cumulative safety constraint violation during training.
weaknesses: - My biggest concern is the effectiveness of neighbor model in step 1, which determines whether a new state-action is safe or not. However, this key component in the proposed shielding method is trained based on data collected in an initial unshielded stage (line 161). During the execution, the policy will be updated and differ from the initial policy, which will lead to a severe distribution shift of state-action pair. Therefore, it's very questionable whether the neighbor model can still distinguish safe state from the unsafe one given unseen new state-action distribution. 
- The construction of safe or unsafe sets $\mathcal S, \mathcal U$ is also problematic. In eq. (3), the feature (state-action pair) is classified as safe if it's start or next state reaches the goal while the feature with next state crashed is unsafe. Such classification obviously introduces some spurious correlations, e.g., the state-action pair is identified as unsafe because it's far from the goal instead of it will truly crash with obstacles. Meanwhile, the contrastive learning objective excludes the inconclusive features, the majority of the collected data, which makes the coverage of training data on state-action space very limited and further exacerbates the distribution shift issue. 
- In experiment, although the results in fig.2 show that ADVICE has smaller cumulative safety violation, ADVICE also performs similarly to baselines on some other tasks (e.g., fig. 8(c) &9). Meanwhile, it's very weird that DDPG-lag and DDPG have very close results (also see questions), suggesting the DDPG-lag baseline is not well-tuned.

Minor issues:
- line 142, distinguish -> distinguishes 
- Eq.(2), $F$ -> $\mathcal F$
- many notations are not defined before using. E.g., $\mathbb Z_+$ in eq. (2), $E$ in line 2 of Algorithm 1.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. In environments with a highly non-smooth reward function, similar features can induce very different rewards (specifically different levels of safety). What exactly is the function $h_\theta$ in (1)? If this function does not relate to safety or reward then the issue of non-smoothness in the safety w.r.t. features may be problematic here.

2. Given that $\mathcal{U},\mathcal{S},\mathcal{I}$ are defined as sets, the construction of $g$ in (3) does not seem like a function since it maps elements of $F_E$ to sets. Perhaps $\mathcal{U},\mathcal{S},\mathcal{I}$ need not be sets for this construction.

3. The construction of $\mathcal{C}$ in (4) also seems spurious since it maps a product of sets to a set which is the union of other sets. Also, it seems not to consider the cases of two dissimilar features in $U$ and secondly, two dissimilar features in $S. 

4. What are $S$ and $U$ in (4) - in page 3, the subindex of $f$ represents time, the indices $i,j \in S$ etc. in (4) therefore seem incorrect.

5. In relation to the construction of the function $g$ in (3), in the black box setting, how can we know which are accepting states?

6. In line 6 of the algorithm, how can we be sure that constructing an action by taking a cartesian product across action dimensions will produce a valid action in the environment? The set of valid actions may only be a subset of this product space e.g. a robot may not be able to lift both its feet off the ground simultaneously.

7. What is the definition of a safety violation - specifically, is there a notion of safety violation that can be used to detect such violations in black-box environments?
soundness: 3
strengths: The methodology is intuitive and the motivation and requirement of each component are reasonably well explained. The problem setting is one of significant interest to the RL community and the approach seems original.
summary: The paper presents a framework to address the problem of safety within unknown environments using a combination of techniques including shielding, contrastive learning and latent feature representations. The framework therefore consists of many components that have been combined under one roof. The target is so called black box environments where the benefit of information relating to safe states is not a priori available so special techniques need to be employed to enable training and execution to proceed as safely as possible. The empirical results show that the framework, ADVICE increases safety violations by a meaningful margin though often at the expense of return
weaknesses: Although some parts of the paper are well explained, especially the non-technical elements and the intuition of the algorithm, many important details are skipped. This leaves the reader to simply guess how important aspects of the framework work. Also, some maths expressions seem to have been written without due care which further adds to confusion. The framework involves various interdependent learning processes - specifically, in addition to the regular policy updates, the action and hence the reward received at a given state depends on the latent representation (which presumably is being updated) as well as the value of K which varies in quite an abrupt manner. This generates concerns about convergence guarantees and the numerical stability of the framework. The paper would benefit from some analysis or at least a discussion on this aspect. Additionally, within the framework there are a number of functions within the method that have max/min functions - this introduces a high level of discontinuity which can be harmful for some learning methods e.g. gradient-based learning methods, and may cause numerical instabilities. 

The novelty of the paper as a whole could be more clearly spelled out. Lastly, although the framework achieves a significant reduction in safety, this often comes at a notable expense to the return. I would like to see how this compares to the frameworks that have some calibration aspect for the performance:safety ratio.

Meta-review: The authors propose a new shielding method based on contrastive learning for safe RL, which learns a latent representation to distinguish between safe and unsafe state-actions. The reviewers all agree to reject the paper, due to weak theoretical results (e.g. unrealistic requirement on diverse data with current safety definition) and weak experimental results (e.g. weak baselines, safety at the cost of performance degrade). Reviewers also mention writing clarity issues. The authors added new theoretical results in the rebuttal, which however warrant another review process. Reviewers also raise concern about the distribution shift issue. Overall, while I think this paper has a potential, the current status needs to be improved.

Decision: Reject