A Distance-based Anomaly Detection Framework for Deep Reinforcement Learning
Authors: Hongming Zhang
Ke Sun
bo xu
Linglong Kong
Martin Müller
Venue: Accepted by TMLR ()

Abstract: In deep reinforcement learning (RL) systems, abnormal states pose significant risks by potentially triggering unpredictable behaviors and unsafe actions, thus impeding the deployment of RL systems in real-world scenarios. It is crucial for reliable decision-making systems to have the capability to cast an alert whenever they encounter unfamiliar observations that they are not equipped to handle. In this paper, we propose a novel Mahalanobis distance-based (MD) anomaly detection framework, called \textit{MDX}, for deep RL algorithms. MDX simultaneously addresses random, adversarial, and out-of-distribution (OOD) state outliers in both offline and online settings. It utilizes Mahalanobis distance within class-conditional distributions for each action and operates within a statistical hypothesis testing framework under the Gaussian assumption. We further extend it to robust and distribution-free versions by incorporating Robust MD and conformal inference techniques. Through extensive experiments on classical control environments, Atari games and autonomous driving scenarios, we demonstrate the effectiveness of our MD-based detection framework. MDX offers a simple, unified, and practical tool for enhancing the safety and reliability of RL systems in real-world applications.

Published in Transactions on Machine Learning Research (10/2024)
A Distance-based Anomaly Detection Framework for Deep Reinforcement Learning
Hongming Zhang"
hongmin2@@uulberto.co
Deportment of Computing Science, University of Alberta Alberta Machine Intelligence Institute (Amát), University of Alberta
Ke Sun"
ksun6@hulberta.car
Department Mathematical and Stalistical Sciences, University of Alberta Alberta Machine Intelligence Institute (Amii), University of Alberta
Bo Xu
boru@ia.com ac.com
Institute of Automation, Chinese Academy of Sciences
Linglong Kong
Ikong@ralberta.ccm
Department of Mathematical and Statistical Sciences, University of Alberta Alberta Machine Intelligence Institute. (Amii), University of Alberta
Martin Müller
mmuulter@uulleert.com
Department of Computing Science, University of Alberta
Alberta Machine Intelligence Institute (Amii), University of Alberta
Reviewed on OpenReview: https: oppereeeepe net/ forun? id- TNKADBV6PA
Abstract
In deep reinforcement learning (RL) systems, abnormal states pose significant risks by poten- tially triggering umpredictable behaviors and unsafe actions, thus impeding the deployment of RL systems in real-world scenarios. It is crucial for reliable decision-making systems to have the capability to cast an alert whenever they encounter unfamiliar observations that they are not equipped to handle. In this paper, we propose a novel Mahalanobis distance- based (MD) anomaly detection framework, called MDX, for deep RL algorithus. MDX simultaneously addresses random, adversarial, and ditoo--distribution (OOD) state outliers in both offline and online settings. It utilizes Mahalanobis distance within class-conditional distributions for each action and operates within a statistical hypothosis testing framework under the Gaussian assumption. We further extend it to robust. and distribution-fres ver- sions by incorporating Robust MD and conformal inference techniques. Through extensive experiments on classical control environments, Atari games, and autonomous driving scenar- ios, we demonstrate the effectiveness of our MD-based detection framework. MDX offers simple, unified, and practical anomaly detection tool for enhancing the safety and reliability of RL systems in real-world applications.
1 Introduction
Deep reinforcement learning (RL) algorithms vary considerably in their performance and are highly sensitive to a wide range of factors, including the environment, state observations, and hyper-parameters (Jordan
et al.] 2020 Patlerson et al.] 2020 The lack of robustness RL algorithms and raised safety concerns
surrounding learned policies hinder their deployment in real-world scenarios, particularly in safety-critical
applications such autonomous driving (Kiran et al. 2021 Liu et al. 2022 Hu et al. 2023) Recently,
TThee authors contributed equally.

---

Published in Transactions on Machine Learning Research (10/2024)
the reliability of RL algorithms has garnered substantial attention (Chan et al. Several studies have highlighted the importance of anomaly detection a crucial safe RL systems (Gareıa & Fernández 2015 Hendrycks et al. 2021] Müller et al. 2022p, emphasizing the
2020 Gu et al. 2024
component for enabling
need for anomaly detection-basce strategics to build trustworthy and safe RL systems (Sedlmeier et al. 2020a; Danesh Fern 2021: Haider el al. 2023
Practical Scenarios. Observed states often contain natural measurement errors (random noises), adver- sarial perturbations, and out-of-distribution (OOD) observations. For instance, consider autonomous vehicle with malfunctioning or unreliable sensors cameras. Under such circumstances, the collected data, such as the vehicle's observed location, can be contaminated by random mcasurement errors. Furthermore, an autonomous car can encounter sensory inputs that have been adversarially manipulated regarding tralfic signs. For example, a stop sign maliciously altered to be misclassified as a speed limit sign (Chen et al. 2019), increases the risk of traffic accidents. Regarding OOD samples, an RL policy trained to drive only on sunny days will struggle with observations from rainy days, which are beyond its trained experience. Such OOD observations
lead to safety violations, performance degradation, and potentially calastrophic failures. All these scenarios highlight the necessity of detecting inaccurate sensor signals from noisy state obscrvations to ensure a vehicle's accurate and reliable operation. Beyond autonomous driving, anomaly detection is critical in many other applications involving sequential decision-making. In healthcare, the RL agent might adjust treatment. recommendations it detects a sudden change in the patient's health condition (Hu et al. 2022). Similarly, detecting fraud and anomalous market states in financial systems is becoming increasingly instrumental in preventing substantial financial losses from market manipulation and fraudulent activities (Hilal al. 2022
Motivating Examples. Fig. noisy location data in the red region (such as GPS coordinate errors), turns right prematurely, risking an accident. Without anomaly detection, the car reacts incorrectly due to the location error. Fig. [(b) highlights how increasing measurement errors, represented by the standard deviation of Gaussian noises, dramatically degrade policy performance. For instance, autonomous cars with RL systems may take sub-optimal or unsafe actions when processing noisy sensory signals in deployment. In addition, incorporating excessive noise during online training (Fig. [cc] can severely impair policy learning and diminish performance. These motivating examples underscore the importance of detecting different types of abnormal states for developing trustworthy RT. systems in real-world applications.
(a) illustrates potential collision scenario where an autonomous car, relying
Our research aims to provide a general framework for applying anomaly detection in deep RL problems, including problem formulation, detection algorithms, and evaluation scenarios. This study contributes to anomaly detection, particularly within the context of safe RI., which falls under the broader research field
350
350
300
910-0.1
-0--0.3
Mean @
250
250
150
150
location
100
50
50
observed noisy location
0
0.01
0.03
0.1
0.3
2000
4000
6000
8000
10000
Gaussian std
Iteration
(a) Unsafe behavior in autonomous driving (b) Performance degradation when (c) Performance degradation when under noisy sensor signals.
noises injected in policy deployment. noises injected during policy learning.
Figure 1: (a) An autonomous car navigates using location data observed from sensors such GPS. Without effective anomaly detection mechanism, inaccuracies malfunctions in these sensors can cause the car to prematurely turn right, leading to collision. (b) and (c): Performance degradation occurs when noisy states are observed in the Breakout environment. Gaussian noises with increasing standard deviations are injected into the state observations during policy deployment, (b) and policy learning (c).
2

---

Published in Transactions on Machine Learning Research (10/2024)
of managing distribution shift in RL; see Section 2 for detailed discussions. Specifically, we strive to develop effective and unified anomaly detection framework for deep RL in both offline and online sellings.
Offline Setting. In this setting, a dataset is fixed without additional online data collection. Given a pre-trained policy, our objective to utilize a fixed dataset to develop a distanco- anomaly detector tailored for pre-trained policy. This detector aims to effectively identify whether state is outlier | ensuring the reliable operation and stable performance of Docision-making systems in deployment.
Online Setting. In this setting, the RL agent interacts with a noisy environment and continuously updates its policy. Our goal is to develop a detection strategy that identifies state outliers, which are outside the RL system's training experience. Removing these outliers can prevent them from interfering with policy training, leading to a robust learning process of RL systems..
Methodologically, we first design an RL outlier detection approach using Mahalanobis Dis- tance (MD) (De Macsschalck et al.} 2000) within a statistical hypothesis test framework and extend it to a robust MD version [uutler al. 1993). These strategies are applied in a parametric manner under the Gaussian assumption for state features in each class, which may not always be accurate in practice. To address this limitation, Gaussian assumption. We empirically investigate the effectiveness of these proposed detection approaches in both offline and online settings across a representative set of RT. environments, including classical control environments, Atari games, and autonomous driving. Our contributions can be summarized as follows:
introduce non-purametrie conformal version of MD detection to relax the
Our primary technical contribution is the design of RL outlier detection strategies based on the con- cepts of Mahalanobis Distance (MD), robust MD, and conformal inference. The anomaly detection strategies are specially developed for deep RL within a hypothesis test framework, accommodating both parametric (Gaussian assumption) and non-parametric (conformal calibration) approaches.
Secondly, in our online setting, our anomaly detection be applied to dynamic datasct, where the RL policy continually improves when interacting with the enviromment. This dynamic setting contrasts with the simpler anomaly detection in supervised learning with static dataset. To address this challenge, we particularly develop moving window estimation and double self- superviie detectors for anomaly detection in the online RL setting.
To our best knowledge, we are the first to conduct a comprehensive study on distance- based anomaly detection in deep RL, covering all typical types of outliers. Our anomaly detectors can simultane- ously identify random, adversarial, and out-of-distribution state outliers. We perform extensive experiments to verify the effectiveness of our proposed methods in both offlinc and online scttings.
2 Related Work
Anomaly Detection in Reinforcement Learning. Anomaly detection has yet to be extensively explored in RL. The connection between anomaly detection and RL was first established in (Müller et al. 2022 however, their work is mainly conceptual and does not propose practical detection algorithms. Change point detection has been investigated in the tabular setting of RL, particularly in environments described as doubly inhomogeneous under temporal non-stationarity and subject heterogeneity (Hu et al.) 2022). They focus on identifying "best data chunks" within the environment that exhibit similar dynamics for policy learning, while our detection focuses on anomaly detection in deep RL scenarios. Prior studics have also probed anomaly detection in specific RL contexts, such as the ofliie imitation learning with a transformer- based policy network (Wang et al.) 2024) and detecting adversarial attacks within cooperative multi-agent RL (Kazari et al. 2023). However, these studies are limited to specific scenarios that do not address general anomaly detection, even in single-agent RL. RL. Sedlmeier et al. 2020bb introduced a simple policy entropy based
'Compared with the classical tasks of policy evaluation and learning offline RI, our offline setting also utilizes fixed dataset but specifically focuses on developing detection methods given fixed policy.
3

---

Published in Transactions on Machine Learning Research (10/2024)
out-of-distribution detector in one-class classification problems. Haider et al. (2023) proposed a model-based method using probabilistic dynamics models and bootstrapped ensembles, but this approach highly relies on the capability of the learned environment model and is also computationally expensive. Unlike the previously mentioned detection methods tailored for specific RL areas, our research aims to further enhance this field by developing a distance-based anomaly detection framework applicable to a broad range of deep RL scenarios.
bistance-based Anomaly Detection. Recently, there has been growth of interest in developing anomaly
detection strategies in deep learning scenarios (Elmrabit et al., 2020 Pang et al., 2021 In image classifica- tion, Mahalanobis distance (MD) was effectively applied by (Lee et al. 2018), who constructed Mahalanobis confidence score by training logistic regression detector using validation samples. This score was evaluated in a supervised way, relying on a validation sel, and thus it is unsuitable for the RL setting. The "lied" covariance assumption used by (Lee et al. 2018), where class-conditional distributions of pre-trained features share the same covariance, was criticized as implansible by (Kamoi & Kobayashi 2020) based on Gaussian discriminant analysis (Klocka et al.) 1980). In contrast, our detection framework MDX avoids the unrcalistic "tied covariance" assumpt.ion by estimating variance for each class using quadratic discriminant analysis. This approach extends linear boundaries to quadratic ones between classes, offering more flexible and accurate detection (Hastic et al. 2009). Additionally, we have developed a distribution-free detection strat- egy using conformal prediction, which eliminates the reliance on the Gaussian assumption and potentially extends applicability across a wider range of practices.
Robust Statistics for RL. Deep RL algorithms inherently face challenges related to instability and di- vergence due to the use of function approximation, bootstrapping, and off-policy learning (Sutton & Barto, 2018). Employing Mahalanobis distance (MD) for anomaly detection can be particularly sensitive during unstable learning phases. The computation of MD is based on Maximum Likelihood Estimate (MLE), which
is susceptible to outliers or noisy data [Rousseeuw & Van Zomeren 1990p. Robust statistics (Huber 2004)
have been developed to address these robustness problems, especially leveraging robust estimation techniques that are not unduly affected by outliers. For example, Robust MD is a robust version of MD that employs robust estimatoss e.g., Minimum Covariance Determinant (MCD) (Rousseeuw 1984 Grübel 1988), for location and covariance estimation (Maronna & Yohai 2014). Our study enhances the understanding of robust statistical approaches' applicability across a variety of arcas in RI, particularly in anomaly detection.
Conformal Prediction and Conformal Anomaly Detection. Conformal anomaly detection (Laxham-
Ishimtsev et al. 2017) is based on the conformal prediction (Angelopoulos et al. Teng et al. 2023 & popular, modern technique for providing valid prediction intervals for arbitrarily machine learning models. Conformal prediction has garnered increasing attention as it can provide a simple, distribution-free, and computationally effective way of tuning the distribution threshold. Tts validity relies on the data exchangeability condition (Shafer & Vovk, 2008), where different orderings of samples are equally likely, but recent studies have verified its applicability in scenarios involving distribution shift (Tibshirani et al. potential of using conformal inference to detect outliers in the context of RL.
mar & Falkman
2011
2021
2019 Barber et al. 2023 and off-policy evaluation (Zhang et al.) 2023b These examples justify the
Distribution Shift in RL. Developing reliable decision-making systems requires effectively addressing distribution shifts in the RL regime. Pertinent research areas include meta RL (Nagabandi et al. 2018 Xu ct al. 2018 Ajay et al. 2022), transfer RL (Taylor & Stone 2009 Parisotto et al.
2015 Zhu ct al. 2023; al. 2022 Anand & Precup 2024
Abel et al.) 2024), and robust generalization in RT. (Boyan & Moore, 1994 Pinto et al. 2017 Zhang al.
Bai et al.] 2024b), continual RL (Khetarpal
2020 While anomaly
detection and these subfields all need to handle distribution shifts to create trustworthy RL systems, our
work specifically focuses on detecting outliers to ensure reliable decision-making within the capacity of the learned policy. This focus distinguishes our study from the other related subfields to tackle distribution
shifts. For a deeper discussion on their differences, please refer to (Müller et al. 2022
3 Background
Markov Decision Process. The interaction of an agent with its environment can be modeled as a Markov Decision Process (MDP), a 5-tuple (S, R, P,7). S and A are the state and action spaces, P: S AxS + [0, 1] is the environment transition dynamics, R x A xx5 S + R is the reward function and 7 (0,1) is
4

---

Published in Transactions on Machine Learning Research (10/2024)
the discount factor. The policy π is continually updated in this online interaction paradigm. Compared to the online setting, recent popular paradigm for reinforcement learning is offline RL (Levine et al. 2020). In the offline setting, RL algorithms utilize previously collected data to extract policies without additional online data collection.
Proximal Policy Optimization (PPO). The policy gradient. algorithm of Proximal Policy Optimiza- tion (PPO) (Schulman et al.) 2017) has achieved state-of-the-art or competitive performance on Atari games [Bellemare et al., 2013) and MuJoCo robotic tasks (Todorov et al. 2012). Typical policy gradi- ent algorithms optimize the expected reward function p (0, so) = E - ( so] by using the policy gradient theorem (Sutton & Barto 2018). Here To is the 6-parameterized policy function. Trust Region Policy Optimization (TRPO) [Schulman et al. 2015) and PPO (Schulman et al. 2017 utilize constraints and advantage estimation to perform the update by reformulating the original optimization problem with the surrogate loss L(0) as:
Th (8t, Areaaa (st, at)
((0)= = Ex [To.ia (st, 0+)
(1)
where Ameua is the generalized advantage function (GAE) (Schulman et al. 2018). PPO introduces clipping in the objective function in order to penalize changes to the policy that make Th vastly different. from Teoia
L CLIP (0) =E min To (st, az) TTaiaa (St, az)
To (st, as) Teai (St, az)
111--,, c,1+e) ) Amosa (s,,a))
E,
Aross (s), cllp
(2)
where is hyperparameter. We use PPO the algorithm testbed to examine the efficacy of our anomaly detection framework. However, our detection methods are general and can be easily applied to other RL
algorithms (Zhang & Yu 2020) such as DQN (Mnih et al.) 2015; Hessel et al. 2018
A3C (Mnih et al.
2016), and DDPG (Lillicrap et al. 2016 Haarnoja et al. 2018 Fujimoto et al. 2018 Bai et al. 2023
Conformal Prediction. Conformal anomaly detection Laxhammar Falkman 201 Ishimtsev et al.
2017 is grounded conformal prediction (Shaler & Vovk 2008 Angelopoulos et al. 2021), which aims construct a confidence band (X) for Y given a random data pair (X, Y) P and a confidence level 1-aa Suppose we have pre-trained model ğ and a calibration dataset (X1, Yı), .... (X,, Y,,) for conformal prediction. We can then compute a predictive interval for the new sample Xn+1 to cover the unseen response Yu+1 by leveraging the empirical quantiles of the residuals |Y; [Xx)) on the calibration dataset. This further leads to valid prediction intervals such that:
P(Yn+1 E (X-a(X)+1) 21 21aa a,
(3)
where the confidence band expected to be as small as possible while maintaining the desired coverage. A fundamental quantity in conformal prodiction is the non-conformity measure, c.g., the residual |Y; ((X,)), which measures how "different. an example is relative to set of examples (Vork et al. 2005
4 Mahalanobis Distance-based (MDX) Detection Framework
For a deep RL agent, acting on anomalous inputs could result in hazardous situations. Therefore, developing suitable anomaly detectors for deep RL agents is particularly important in safety-critical scenarios. Fig. 2 illustrates the operational flow of our MDX framework.
Description of Detection Framework. Our detection framework is structured around two core com- ponents: feature extraction and detector estimation. The process begins by assessing whether state is anomalous, which is crucially dependent potentially unsafe action is labeled as an outlier. Specifically, we input the state into the policy network and extract the feature vector from the penultimate layer of this network. We calegorize states according to the actions determined by the policy, based on the intuition that states associated with the same action share similar features. For each action class, we estimate the mean value (μ) and covariance matrix (E) of the feature vectors as the class centroid. A threshold is set as the class boundary that partitions the feature space into inliers and outliers.
the associated policy. A state that prompts the policy to initiate
5

---

Published in Transactions on Machine Learning Research (10/2024)
$
outlier
Conv/FC layers
(U 2nn
(,
FC layer
feature
FC layer
(f()).c
action class
Tt- Is)
(122 22)
inlier
Figure The detection pipeline of MDX. We feed the state into the policy network to extract the feature vector and identify its class. For each class, we estimate (H, Σ) and establish a detection threshold depicted dashed ellipse. To determine whether new state is outlier, we evaluate its features and compute the distance to the class controids. If the distance falls below the set threshold, the state is classified as inlier (green points). Conversely, the state is marked as an outlier (red points).
After the class centroids are estimated, we can determine whether a new state an outlier by computing its distance from the established class centroids using the Mahalanobis distance based on its feature vector. The Mahalanobis distance is measure of the distance between point and distribution, which in this case is represented by the class centroids. A stale is deemed an outlier if the distance surpasses the predefined threshold. An appropriate threshold can balance the trade-off between false positives and false negatives, ensuring that the detection system is both accurate and reliable. By ensuring that only states within the policy's capability are considered valid, MDX enhances the safety and reliability of the RL system.
Our detection framework is generic and can be applied to any agents that operates based on learned neural network. Based on the proposed detection framework, we instantiate two detection methods in Section distribution-bass detection under the assumption of Gaussian distribution, and distribution-fee frree detection, which is based on conformal prediction. The former method relies Chi-square distribution to determine the threshold, while the latter method employs the conformity score to establish the threshold. We then extend the detection framework to the online setting in Section G the agent interacts with the environment.
where the detectors are updated continuously as
5 Anomaly Detection in the Offline RL Setting
Our MDX detection framework mainly induces two kinds of detection algorithms, including distribution- based detection under Gaussian assumption in Section 5. and distribution--ree detection by leveraging conformal prediction Section 5.2 Finally, an integrated detection algorithm pipeline provided, incor- porating all variants of detection strategies in the RL context.
5.1 Distribution-be based Detection under Gaussian Assumption
Gaussian Assumption. The given pre-trained parameterized RL policy Te a discriminative softmax classifier, Taa = cls;) exp (w: f(s) b.) Ee exp (wT + bz), where we and b are the weight and bias of the policy classifier for action class The function f(-) represents the output of the penultimate layer of the policy network ne, serving as the state feature vector. Here, C' |A| is the size of the action space, and Re is the mean vector of f(s) corresponding to the action class c A If we assume that the class-conditional distribution follows multivariate Gaussian distribution sharing a single covariance Σ (tied covariance) in a generative classifier, i.c., (f( a c) = N (f(s) Me, E), then the posterior distribution of f(s) matches the form of a discriminative softmax classifier (Lee al. 2018). This equivalence implies that f(s) fit a Gaussian distribution under Tg. We approximate state feature vectors with a class- cans-conniii Gaussian
2Our MDX detection framework currently focuses environments with the discrete action spaces. For continuous action spaces, natural solution discretize the actions into several bins and then follow the same detection pipeline, which deserves further validation.

---

Published in Transactions on Machine Learning Research (10/2024)
distribution with Me and Eg for each action class, rather than using single "tied" covariance Σ across all action classes (Kamoi & Kobayashi 2020).
Vanilla MD-based Detection. An MD-based detection based on Gaussian assumption can be immediately developed based on the mean vectors Pe and the covariance matrix Ee calculated from f(s) for each action class c. We first collect N₂ state action pairs ((8,,0;)) separately for each action class e, and compute the empirical class mean and covariance of
Pe
1 No Σ f(s),
E.
1 (f No =0,=0
(4)
In distance- bissance-based detection, a straightforward metric is Euclidean distance (ED). However, MD generally outperforms ED in many tasks (Lee ct al.) 2018 Kamoi & Kobayashi 2020 Ren et al. 2021 as it incorpo- rales the additional data covariance information to normalize the distance scales. Following the estimation in Eq. we derive the consitonni Gaussian distribution to characterize the data structure within the state representation space for each action class. For each state s observed by the agent, we compute its Detection Mahalanobis Distance ((s) between 8 and the nearest class-conditional Gaussian distribution by:
M(s) = min (f(s) - R2o) 5 (f(s) Pa))
(5)
Unlike the previous work Lee et al. (2018), which defined a Mahalanobis confidence score based on a binary classifier in a validation dataset, we utilize (s) as the detection metric within statistical hypothesis test
framework. Proposition demonstrates that M(s) follows Chi-squared distribution under the Gaussian assumption.
Proposition 1. (Test Distribution of Detection Mahalanobis distance M(s)) Let f(s) be the p-dimensional stute random vector for action class c. Under the Gaussian ussumplion ((](s)la c) =N (J(ss) Mey Ec), the Detection Mahalanobis Distance M(s) in Eq. (5 is Chi-Square distributed: ( ~ x
Please refer to Appendix A for the proof. Based on Proposition we can define a threshold e = X² - a) by selecting a a value from the specified Chi-Squared distribution to distinguish normal states from outliers. Given a new state observation 8 and a confidence level a, M(s) > O is detected as an outlier.
Robust MD-based Detection. The estimation of Me and D. in Eq. relies on Maximum Likelihood Estimate (MLE), which is sensitive to the presence of outliers in the dataset (Rousseeuw & Van Zomeren, 1990h. As the offline data collected from the environment tends to be noisy, directly introducing MD for outlier detection in RL easily results in a less statistically effective estimation of and E.; thus undermining the detection accuracy for outliers. This vulnerability the MDDbaacd detector against noisy states prompts us to instantiate MDX with a more robust estimator (Huber 2004).
To this end, we apply the Minimum Covariance Determinant (MCD) estimator (Hubert Debruyne 2010 to estimate Pe and E. by only using subset of all collected samples. It only uses the observations where the determinant of the covariance matrix is small possible. Concretely, MCD determines the subset J of obscrvations with a size h, while minimizing the determinant of the sample covariance matrix calculated
Hun
(a) Random Outliers.
(b) Adversarial Outliers.
(c) OOD Outliers.
Figure 3: Contours under the estimation based on MD and Robust MD across different outlier types on
Breakout, Black and red points denote inliers and outliers, respectively. The dimension state feature vectors after a pro-trained PPO policy is reduced by t-SNE (Van der Maaten & Hinton 2008).

---

Published in Transactions on Machine Learning Research (10/2024)
solely from these h points. The choice of h determines the trade-off between the robustness and efficiency of the estimator. The robust MCD mean vector pro and covariance matrix Drob in the action class c are computed as
Prob
((6), J J={ set points E> S Ex for all subsets K]. BEJJa.
Σ
(6)
where we set as (number_en of _aa_bee___saappesssss______see + number_ of features + 1)/2 (Rosseeuw 1984). K represents the total number of subsets that contain h points. In practice, the MCD estimator can be efficiently solved by the FAST-MCD algorithm (Hubert & Debruyne 2010) instead of performing brute-force search over all possible subsets. Akin to Mahalanobis Distance, define the Detection Robust Mahalonobis Distance (s) as robust detection metric:
(s) min (f(s) - p2ob) Errob- (f(s) - p2bb). ((())~peb)
Since the robust Mahalanobis distance can still approximate the true Chi-squared distribution (Hardin & we still employ the threshold value θ = x -a) a) for detecting outliers as in the MD case.
Rocke 2005
As a motivating example, Fig. 3 displays contours computed by both MD and Robust MD detection methods for state feature vectors in the Breakout game from the popular Atari benchmark (Bellemare al.j 2013 Brockman al., 2016) with different lypes of outliers. These results demonstrate that estimation based on Robust MD is less vulnerable to outlying states (red points) and better fits inliers (black points) than MD. This robust parameter estimation highlights the potential advantage of Robust MD for RL outlier detection, where the data used for estimation tends to be noisy.
5.2 Distribution-free Detection by Conformal Inference
Although robust MD-based detection is less vulnerable to noise RL environments, both MD and robust MD strategies heavily rely on the Gaussian assumption to construct the detection thresholds based on Proposition ! This distribution assumption is often violated practice, diminishing the effectiveness of MD and robust MD. In contrast, conformal prediction offers mathematical framework that provides valid and rigorous prediction distribution without assuming specific underlying data distribution. The resulting conformal anomaly detection circumvents the limitation of the distribution assumption, potentially improving the detection efficacy.
In the context of RL, conformal anomaly detection evaluates how a state conforms to a model's current prediction distribution, thereby discriminating abnormal states. As distribution--ree detection approach, conformal anomaly detection can enhance the ristance-based detectors by additionally tuning the anomaly threshold in the calibration dataset. To design the conformal anomaly detection method, we leverage the Detection Mahalanoibis Distance M(s) the non-conformilly score, which measures how dissimilar state is from the instances in the calibration set. Following split conformal inference (Papadopoulos et al. 2002 Shafer & Vovk 2008), we split the the previously collected offline dataset into the the calibration set Deal and the evaluation set. A simple way is to evaluate the quantiles of the resulting empirical distribution to create the corresponding confidence band. Using the calibration set Deal, we define the fitted quantiles Qi- of the conformity scores for the action class e as follows:
( { g: N
(
ii = inn
inf
Σ s.EDEal,a
1MM 2 21-0
where each (8t, a4) is drawn from the calibration set Deal and e is calculated by c= arg min (x)))) in M* among all action classes. Finally, we use the class-dependent and well-calibrated detection thresholding θ Qi-a in conformal MD-based detection instead of x²(1 - a) used MD and Robust MD strategies.
5.3 An Integrated MD-based Detection Algorithm in the Offline Setting
8

---

Published in Transactions on Machine Learning Research (10/2024)
Algorithm 1 MDX Detection Framework in the Offline Setting
Input: The given policy иe, the dimension of state feature vectors p, and a confidence level a. Output: Detection labels {ya} for each in the evaluation trajectory.
3: /
Step 1: Detection Design by Estimating Mean and Covariance
4: Given state action pairs {(8,,0;) where d; ~ no(-(s;).
5: for each action class с do
6: if we choose MD detection then
7: Estimate Te and E. via Eq. (4 /* Approach MD Detection * */
else if we choose Robust MD detection then Estimate prob and Egob via Eqs. (6) and
"Apprrach 2: Robust MD Detection * */
10: else
11: Estimate Be. Ex via Eq. 04
calibrate Qi-a via Eq. 8
Approach 3: Conformal MD-based
Detection */
12: end
13: end for
14: /* Step 2: Detection Deployment *
15: for in the noisy environment do
16: Compute distance d (s) or d Ms)b(s), and threshold θ = xx1(1 a) θ = Qi-a'
17: Set Detection label y, if d > θ clsc y, = 11
18: end for
In practice, it unclear whether the collected data is noisy or to what extent the Gaussian assumptioo is violated. Therefore, we provide an integrated algorithm pipeline. Algorithm [ summarizes all the variant detection strategics of MDX in the offline setting. We compute the (robust) mean vector and covariance matrix among the state feature vectors in the penultimate layer of for each action class. Next, given a state observation s, we compute the detection Mahalanobis distance d = ((s) or d = Mrob(s) and compare it with the threshold θ x a:) under the Gaussian assumption θ Qi from distribution-free conformaa quantiles. If d O, is detected as outlier. Conversely, if d < O is identified
inlier.
6 Anomaly Detection in the Online RL Setting
In the online RL setting (Sutton & Barto 2018 Dong et al. 2020), policy is updated continuously, unlike the fixed pre-trained policy used in our offiine setting. Robust policy training with noisy states is crucial in safe RL, as the agents are more likely to encounter state outliers during training. In this section, we extend MDX to the online RL training scenario. Unlike the offline setting, the challenge here stems from the dynamic nature of policy updates, requiring our detector to adapt to the evolving distribution of feature vector outputs. The complexity increases when the improved policy starts gathering new samples through exploration, posing fundamental challenge in online RL framework. An effective detection system must differentiate between actual noisy observations and newly collected data through exploration. Training the RI. agent and estimating the detector are interleaved in noisy online environment. Various options for managing detected outliers during training include removing or denoising the outlier states. In our detection framework, we focus on direct removal and assess the resulting learning curves in the presence of noisy states during the training process. To address the challenges in detecting abnormal states in the online training setting, we propose Moving Window Estimation and Double Self-supervised Detectors, both of which are pivotal for the empirical success of our anomaly detection approach.
Moving Window Estimation. In the online setting, improving the policy mo causes shift in the data distribution within the replay buffer as the agent interacts with the environment (Rolnick et al. 2019 Xiao et al. 2019). To effectively utilize information from the updated data distribution, we maintain a moving window to store experiences throughout the interaction steps. The moving window operates like a first-in- first-out buffer, storing the most recent samples and discarding the oldest ones. Data falling outside the window cast away. The moving window can be adjusted either prioritize long historical context with larger window size consider only more recent experiences with smaller size. In our experiments, since
9

---

Published in Transactions on Machine Learning Research (10/2024)
Algorithm 2 MDX Detection Framework in the Online Setting, PPO Style
Initialize policy network vg and estimator The and E
Initialize confidence level - a, the moving window size m, inlier and outlier buffers B1, Bo 3: for iteration = 2, ..., K do
4: for actor = 1, ..,,N N do
5: Run policy wo in environment for T timesteps.
6: Compute distance d = (s), and threshold e = x²(1 1p(1-10 - a) or = Qi-aa if d < then
Add it to B,. else
10: Add it to Bo. end if
11:
12: end for
13: Optimize policy me using inlier trajectories.
14:
Update Th and Eg based on data in the moving window every N, new samples come.
15:
end for
the environments we consider have finite horizons with restarts, catastrophic forgetting is not concern. We set small window size of 5120 to balance past and recent data used for detection estimation. Based on the constantly updated state feature vectors, Ae and Eg are continually estimated. This continuous updating allows us to accurately track the state feature distribution, ensuring that our detector remains sensitive to recent and historical data shifts.
Double Self-Supervised Detectors. Our current detector is continually refined using self-detected inliers, while any detected outliers promptly discarded. However, more practical approach is to leverage these outliers to create a complementary detector for outliers. This sccondary self-supervised detector validates the detection results from the primary detector. For example, if the primary detector classifies a state as an inlier and the secondary detector agrees that it is not an outlier, the state confidently classified as such. Conversely, if there a difference between the discrimination of the two detectors, the state is randomly classified as either an outlier or an inlier. In the event of disagreement, this random classification is motivated by the need to avoid systematic bias that could arise from consistently favoring one detector's output over the other. By introducing randomness, we ensure the system remains fair and does not overly rely on potentially flawed outputs from either delector. This approach also preserves the system's 's ability to learn and adapt over time, preventing the reinforcement of incorrect classifications. The double-detector system thus enhances the robustness and reliability of the detection process, ensuring more accurate and consistent identification of abnormal states.
MD-based Detection Algorithm in the Online Setting. Algorithm 2 outlines our MD-based detection procedure for online RL, incorporating both moving window estimation and double sclf-supervised detectors. To update our double detectors, inliers and outliers are stored in buffers B, and Bo, respectively. For each class, window size is specified. Within each class, the state-action pairs in the window are used to estimate the and E.. These parameters are updated after every N, newly collected data points in the window for action class c. This adaptive updating mechanism ensures that the detectors remain responsive to evolving data distributions.
Online Anomaly Detection Procedure. Since our detection method relies on features extracted from the penultimate layer of the policy network, instead of training the policy from scratch, we pretrain the policy to ensure these features capture meaningful information about the environment. This pre-trained policy results in more mcaningful state features and enhances the detection procedure, contributing to a rapid assessment among distinct detection algorithms through their learning curves. Moreover, deploying a randomly initialized policy in a real-world scenario is unreliable. Instead, it is common practice to use a pro-trained policy as warm start and then further improve For example, in recommendation systems, a pre-trained policy is deployed initially to provide recommendations, and user feedback, such as click-through rates (CTR), is used to iteratively update the online policy. Similarly, within our online detection algorithm,
10

---

Published in Transactions on Machine Learning Research (10/2024)
we pre-train a policy using inlier data a warm start. After pre-training, the policy is introduced to the noisy environment for further online learning. Throughout this process, our MDX framework is used to identify outliers in the subsequent training phases. We then evaluate the training performance of algorithms equipped with these detection mechanisms. This systematic approach facilitates the gradual refinement of the policy while concurrently integrating outlier ion to enhance robustness in real-world settings.
7 Experiments
We first conduct experiments on both feature-input and image-input tasks to verify the effectiveness of our MDX framework in both offline and online settings. For feature-input tasks, we choose two classical con- trol environments in OpenAI gym (Brockman et al.) 2016). including Mountain Car (Barto et al., 1983) and Cart Pole Mooore] 1990). For image-input tasks, we choose six Atari games (Bellemare et al. 2013 We divide the six Atari games into two different groups. The first group includes Breakout, Asterix, and Spacelnvaders, which feature nearly static backgrounds. Enduro, FishingDerby, and Tutankham in the sec- ond group have time-changing or dramatically different backgrounds, presenting more challenging scenarios. We further conduct experiments on autonomous driving environments (Dosovitskiy et al.] 2017) as one po- tential application. We select Proximal Policy Optimization (PPO) (Schulman al.] (017)) as our baseline RL algorithm. For feature-input classical control tasks, we policy network with two fully connected layers, each containing 128 units with ReLU activation functions. For image-input tasks, we use the same network architecture as described in the PPO paper (Schulman et al. 2017).
Three Types of Outliers. (1) Random Outliers. We generate random outliers by adding Gaussian noise with zero mean and different standard deviations on state observations, simulating natural measurement errors. (2) Adversarial Outliers. We perform white-box adversarial perturbations (Szegedy ct al. 2013
Goodfellow el al. 2014b; Cao et al. 2020) on slate observations for the current policy, following the strategy proposed in (Huang et al. 2017 Pattanaik et al. 2017). Particularly, we denote af as the 'worst" action, with the lowest probability from the current policy Te (a(s). The optimal adversarial perturbation l/t, constrained in an e-ball, can be derived by minimizing the objective function J: min, [[s - 77, Tt) = =- -En P log Te (a, n), 8.11|lll S 6, where p'u. = and pf = for i * w. We solve this minimization problem with the Fast Gradient Sign Method (FGSM) (Goodfellow et al. 2014b), a typical adversarial attack method in the deep learning literature. The resulting adversarial outlicrs s, + 17," force the policy to choose a'..- (3) Out-of- Distribution (OOD) outliers. OOD outliers arise from the disparity in data distribution across different enviromments. To simulate them, randomly select states from other environments and introduce them to the current environment. In our experiments, select images from other Atari games to serve as Out- of- Distribution (OOD) outliers within the considered environment. In the autonomous driving scenario, we designate rainy and nighttime observations as OOD outliers for the primary daytime setting on sunny day. This deliberate selection of diverse outlier examples enables comprehensive testing of our mothod's robustness across varied environments.
Baseline Methods. A fundamental obstacle in assessing the anomaly detection strategies in RL lies in the scarcity of suitable basclines in deep RL settings introduced in Section 2 To rigorously substantiate the effectiveness of MDX, we initiate our evaluation by comparing them with the foundational baselines we have developed ourselves and implement two non-distance-basse methods. (1) Euclidean distance (ED) assumees that all features are independent under the Gaussian assumption with one standard deviation, which can be considered as a simplified version of our MD method with an identity covariance matrix. (2) MD with Tied covariance (TMD) follows the tied covariance assumption (Lee et al., 2018), where features among all action classes share a single covariance matrix estimation. (3) PEOC is policy entropy-based detection method proposed in Sedlmeier et al. (2020b). The authors assume that successful training process reduces entropy for states encountered during training, which can then be used as classification score to detect OOD states. (4) EnvModel follows the model-based detection algorithm utilizing lcarned dynamics models and bootstrapped ensembles (Haider el al. 2023). We Irain five auloencoders as environment transition models in each environment. For the offline setting, the autoencoders are trained based on the dataset, while for the online sctting, they are continuously updated. Each autoencoder predicts the next state given the current state and action, and the minimum prediction error among the five models serves as the anomaly detection signal. (5) MD is our first. proposed method with class-conditional Gaussian assumption. (6)
11

---

Published in Transactions on Machine Learning Research (10/2024)
Detection Accuracy (%)
Outliers
ED
TMD
PEOC
EnvModel
MD
RMD
MD+C
Random
68.0
93.9
50.0
50.0
95.4
78.9
96.2
Cartpole
Adversarial
51.1
93.2
50.0
50.0
94.5
78.7
94.8
OOD
87.3
94.3
50.0
93.8
96.5
79.1
97.5
Random
89.5
86.4
50.0
49.9
90.6
78.1
93.6
MountainCar
Adversarial
64.0
81.0
50.0
49.7
85.4
74.3
87.1
OOD
90.9
86.5
50.0
48.7
90.5
77.2
91.7
Random
78.8
90.1
50.0
50.0
93.0
78.5
94.9
Adversarial
57.6
87.1
50.0
49.8
89.9
76.5
90.9
Average
OOD
89.1
90.4
50.0
71.3
93.5
78.2
94.6
Average 75.1
89.2
50.0
57.0
92.1
77.7
93.5
Table I: Average detection accuracy of MD, RMD, and MD+C compared with baselines across different outlier types in feature-input classical control environments in the offline setting. The averages are computed across environments and outlier types. Accuracy is determined by applying detection techniques to the balanced data composed equally of clean and noisy states.
Robust MD (RMD) is the robust variant of MD under the Gaussian assumption. (7) MD+C uses well- calibrated conformality scores to construct a valid empirical distance distribution instead of relying on the Chi-Squared distribution established upon the Gaussian assumption.
7.1 Anomaly Detection in the Offline Setting
In the offline setting, we randomly split the states from the given dataset into calibration and evaluation sets, each containing 50% of the data. The calibration set is used to construct our detectors, and the evaluation set is for testing. We first use PCA to reduce the state feature vectors into a 50-dimensional space. We then apply (robust) MD to estimate mean vectors and covariances and calibrate the conformality score based on
Detection Accuracy (%)
Outliers
ED
TMD
PEOC
EnvModel
MD
RMD
MD+C
Random
53.2
59.1
50.0
50.0
61.4
71.2
62.8
Breakout
Adversarial
84.3
89.1
50.0
50.0
90.8
80.2
91.7
OOD
56.9
47.8
50.0
49.6
79.5
50.8
Random
43.9
45.1
50.0
50.0
60.3
69.5
54.7
Asterix
Adversarial
83.7
85.6
50.0
50.0
75.2
94.0
0OD
39.6
40.8
50.0
53.8
46.1
57.7
49.7
Random
51.4
63.9
50.0
70.2
79.4
68.6
SpaceInvader
Adversarial
70.9
90.3
50.0
50.0
96.1
81.1
96.5
OOD
45.3
45.9
50.0
57.0
81.0
53.6
Random
49.0
59.0
50.0
50.0
72.7
82.4
70.2
Enduro
Adversarial
92.9
91.3
50.0
50.0
96.2
83.3
97.5
0OD
57.1
74.9
50.0
47.6
80.0
83.4
63.3
Random
18.9
66.1
50.0
50.0
69.8
85.7
66.5
FishingDerby
Adversarial
86.3
92.5
50.0
50.0
97.4
87.2
97.4 4
OOD
51.3
56.2
50.0
61.7
81.5
58.5
Random
50.0
47.5
50.0
50.0
49.1
74.0
49.8
Tutankham
Adversarial
66.2
89.5
50.0
50.0
95.3
77.1
96.5
0OD
55.0
83.3
50.0
97.5
89.6
77.2
79.7
Random
49.4
56.8
50.0
50.0
63.9
77.0
61.7
Adversarial
80.7
89.7
50.0
50.0
94.6
80.7
95.6
Average
OOD
50.8
58.1
50.0
67.8
63.6
76.7
59.3
Average 60.3
68.2
50.0
55.7
74.0
78.1
72.2
Table 2: Average detection accuracy of MD, RMD, and MD+C compared with baselines across different outlier types in six Atari games in the offline setting. The averages computed across environments and outlier types. Accuracy is determined by applying detection techniques to the balanced data composed equally of clean and noisy states
12

---

Published in Transactions on Machine Learning Research (10/2024)
the calibration dataset. Finally, incorporate the three types of noises into the originally clean evaluation dataset. We assess the performance of our detection methods on the entire evaluation dataset.
Main results. Tables I and 2 show the detection accuracy of MDX instantiated with vanilla MD, robust MD, and conformal MD with a = 0.05 across a wide range of outlier types on each task. A higher accuracy indicates a more successful identification of anomalies for the evaluated detection method. We conclude that: (1) All MD-based methods, ie., TMD, MD, RMD, and MD+C, outperform ED, confirming the usefulness of covariance matrix information in RL outlier detection. (2) MD+C performs consistently best classic control tasks and excels in identifying adversarial outliers Atari games. Robust MD generally performs the best on Atari games, significantly surpassing MD and other methods in detecting random and OOD outliers. Nonetheless, robust MD is not effective enough to detect adversarial outliers. We hypothesize that the robustness advantage resulting from RMD in detector estimation is more applicable in image imput or the high-dimensional state space. (3) PEOC is almost ineffective across all considered state outliers, suggesting that the entropy difference useless for identifying outliers. By contrast, EnvModel is only superior to the other approaches against some OOD outliers, which not generally preferable. More detailed results are provided in Tables and 8 of Appendix B.1
Sensitivity Analysis on Feature Dimension Reduction. We provide a sensitivity analysis on Atari games regarding the number of feature dimensions reduced by PCA, showing that the detection accuracy for all considered outliers tends to improve as the number principal components increases. This indicates that better detection performance can be achieved with higher feature dimensions. The detailed results are presented in Appendix B.4
Effectiveness of Robust MD. In robust MD analysis, it is typically concluded that outlier states distinctly separated from inlier states. By comparing the Mahalanobis distance distributions between inliers and outliers under both MD and Robust MD, we show that this conclusion also applies to the RL anomaly detection scenario. This effect explains the detection advantage robust MD in RL. Detailed results are provided in Appendix B.3
7.2 Anomaly Detection in the Online Setting
The PPO agent, utilizing multi-processes as detailed in the original PPO algorithm (Schulman et al. 2017); runs eight independent environments in parallel, and introduce state outliers into four of these envi- ronments. For random and adversarial outliers, actions are determined based on the PPO policy network
-12P
Maz
5200
-440
-148
Moan
-104
Meen
-180
-20
-281
-253
38
1/2
1/2
1A
Step
Step
S2ep
Step
JMD
Auto
Elytoode
RACTOR Porforancc
-129
5D Bivioaoe
NDrC
-208
VD+C
JND
*D
MD
Auto
/Eoc
TMQ
-288.
-EECC
EnwMedel
+130
112
-115:
PIcc
THD
Fandam
Rundam
Yundom 55
Random
05 13
025
27
33
12
33
Gaussian std=1
(b) Gaussian std=0.3
(e) OOD CartPole
(d) Adversarial
Figure 4: Performance on MountainCar across various state outliers in online learning. The first row shows the policy performance during learning. The second row shows the relationship between the averaged detection accuracy and performance.
13

---

Published in Transactions on Machine Learning Research (10/2024)
16.
389
106
264
377.
140
190
330
Enration
Eeeatton
(teration
Benston
IrsWooo
Auti
/Aula
KDIC
380
ND
Acce
.40
1.5
tyf
315-
PEOC "IND
MD
305
EnsMode
143
145
50
3910
FD
TND
pardom
151
Zancom
AND
TWQ
159
Sandom
Bandom
02
Score
Score
Scon
P1 Sore
(a) Gaussian (std=1) (b) Gaussian (std=00.3
(c) OOD Enduro
(d) OOD FishingDerby
(e) Adversarial
Figure 5: Performance on Tutankham across various state outliers in online learning. The first row shows the policy performance during learning. The second row shows the relationship between the averaged detection accuracy and achieved final performance.
wo- For OOD outliers, due to the potential differences in action spaces between the original environment and the OOD environment, we select OOD states from the OOD environment by taking random actions within its own action space. For the Robust MD method, we use PCA to reduce state feature vectors into a 50-dimensional space due to the expensive computation of the robust MD method. For the other methods, we use the original feature vectors output from the penultimate layer of πо- Results are averaged over three seeds with hyperparameters given in Table 7 of Appendix [C.]] When our detectors identify an outlier, it is removed from training. We compare the resulting learning curves for different detection methods.
Additional Basclines. We add another two basclines performance upper bound and lower bound. (1) For an ideal baseline, the method Auto automatically deletes true state outliers, showing the optimal training performance of algorithms without the interruption from outliers.
(2) At the other extreme, Random uses a totally random detector that detects a state as an inlier or outlier with probability of 0.5.
Main Results. Figs. andd 5 present the online performance on feature-input task MountainCar and image- input Atari game Tutankham. The first row shows the learning curves of cumulative rewards based on the PPO algorithm. To better highlight their differences, full results with confidence bands in Appendix C.I Figs. 1 to 21 for reference. The second row illustrates the relationship between detection accuracy and policy performance. For Atari games, the x-axis represents the average F1 score during learning, while the y-axis represents the final performance. For featuro-input tasks like MountainCar and Pole, some methods can achieve the maximum score, leading no significant difference in final performance. We exhibit the relationship between the average F1 score and the average
omit the confidence bands in Fig. 5 while providing
Superiority Rank
Outlier Type
Random
ED
TMD
PECO
EnvModel
MD
RMD
MD+C
Random
6.75
3.75
7.25
6.00
3.25
4.50
3.25
1.25
Performance
OOD
7.75
6.25
4.25
5.00
1.50
3.75
4.50
3.00
Advensarial
7.00
6.00
5.00
8.00
1.50
3.50
2.50
2.50
Average
All
7.20
5.20
5.60
6.00
2.20
4.00
3.60
2.20
Random
8.00
4.75
5.75
6.00
3.00
4.25
3.25
1.00
Score
OOD
8.00
5.75
4.75
5.75
4.25
4.00
1.75
Advensarial
8.00
5.50
5.50
7,00
1.00
3.00
3.00
3.00
Average
All
8.00
5.30
5.30
6.10
2.10
4.00
3.50
1.70
Table 3: The average superiority rank (1 is best)) of anomaly detection methods across all types of outliers. Numbers in bold represent the best results.
14

---

Published in Transactions on Machine Learning Research (10/2024)
policy performance during learning, similar to previous work (Zhang et al. 2023a). We can find that higher detection accuracy is generally associated with better policy performance. For each outlier type in Table B. we evaluate the superiority rank of all detectors regarding the F1 score and policy performance, where rank 1 indicates the best performance. A smaller superiority rank implics a more effective detection. Our conclusions are as follows: (1) Conformal MD (MD+C) generally achieves the best detection performance across all considered baselines (except Auto). The superiority of MD+C over MD highlights the crucial role of accurately calibrated thresholds in the online RL detection setting. (2) PECO is ineffective in most of our experiments, which indicates that policy entropy is not a reliable indicator for detecting anomalies. (3) While EnvModel achieves competitive performance in most tasks, it is less practical because it requires executing the action in the environment to obtain the next state before detecting the current state. That being said, EndM is a hindsight method and is less applicable in real applications.
A Allation Study on Double Self-Supervised Detectors. We conduct an ablation study of double self- supervised detectors on Breakout with random and OOD outliers. Results in Fig. 22 of Appendix C.2 show that double self-supervieed detectors reduce detection errors and improve detection accuracy.
Ablation Study Outlier Proportions. We also demonstrate the robust detection performance across different proportions of outliers encountered by the agent during training. We conduct experiments on Breakout, and the results are provided in Fig. 23 of Appendix C.3
7.3 Autonomous Driving Environment
To verify the broader applicability of our method, we perform experiments autonomous driving environ- ments based on CARLA (Dosovitskiy et al., 2017) and introduce practical scenarios in which all three types of anomalies commonly occur. Since anomaly detection in autonomous driving is more practical in the offline setting and CARLA is a complex environment that exceeds our computational capacity for online training, we focus on the offline setting in this section.
Random Noise. Malfunctioning sensors or cameras can introduce random noise into signal observations. For instance, faulty camera lens may produce distorted images, while a malfunctioning LiDAR sensor might generate erroncous depth measurements. Such random noise can impair the reliability of perception systems in autonomous vehicles.
Adversarial Attacks. Adversarial attacks involve intentionally manipulating input signals to disrupt the functioning of RL systems (Bai et al.] 2024a). In the context of autonomous driving, an attacker might tamper with sensor data or traffic signs, resulting in misleading observations and potentially hazardous driving bchavior. Adversarial states thus pose a significant threat to the robustness and safety of autonomous driving systems.
Out-of-Distribution (OOD) States. Consider scenario where an RL policy is trained exclusively under sunny weather. Encountering rainy weather poses challenge, as the observations captured under these conditions deviate from the training data distribution. Such observations are therefore considered Out-of- Distribution (OOD) states.
(a) Original state
(b) Random outlier
(c) Adversarial outlier (d) OOD outlier (rainy) (e) OOD outlier (night)
Figure 6: The clean and noisy state observations in autonomous driving experiments.
15

---

Published in Transactions on Machine Learning Research (10/2024)
Detection Accuracy (%)
ED
TMD
PEOC
EnvModel
MD
RMD
MD+C
Random (std E [0.005, 0.07)
50.0
62.0
50.0
50.0
68.7
72.1
60.7
Random (std E (0.07, 0.3])
50.0
95.4
50.0
50.0
95.2
73.8
95.8
Adversarial
50.0
96.4
50.0
50.0
91.0
73.8
97.5
OOD (Rain)
50.0
96.4
50.0
52.7
95.3
73.6
97.5
OOD (Night)
50.0
96.4
50.0
97.5
95.3
73.8
97.5
Table Detection accuracy on the CARLA town environment over three types of outliers.
Experimental Sctup. We conduct experiments using the CARLA environment (Dosovitskiy et al. 2017 CARLA is open-source simulator for aulonomous driving research known for its high-quality rendering and realistic physics. The environment includes 3D models of static objects, such as buildings, vegetation, traffic signs, and infrastructure, as well as dynamic objects, such as vehicles and pedestrians. The task is to drive safely through the town. In each episode, the vehicle must reach a given goal without collision. The episode ends when the vehicle reaches the goal, collides with an obstacle, or exceeds the time limit.
Noisy State Observations. Following the approach used in Atari game settings, we introduce Gaussian noise to simulate random outliers and generate adversarial outliers using adversarial perturbations. For OOD outliers, we leverage CycleGAN-Turbo (Zhu et al. 2017: Parmar et al., 2024), technique designed for adapting a single-step diffusion model (Ho et al. 2020) to new tasks and domains through adversarial learning (Goodfellow et al.) 2014a). This method can perform various image-to-image translation tasks and outperforms existing GAN-bases and diffusion- methods for various scene translation tasks, such as day-to-night. conversion and Specifically, we use CycleGAN-Ture to create rainy and nighttime outliers. Examples of dilferent anomaly states presented in Fig. 6
weather effects like fog, snow, and rain (Parmar et al., 2024)).
Main Results. Given a fixed dataset and a pre-trained policy, we assess our dctection methods across the three lypes of outliers. Table shows the average accuracy, with MD+C achieving the highest performance in most scenarios, while RMD performs best in the presence small random noises. Similar to the results in classical control environments and Atari games, the entropy- based PEOC is still ineffective across all outlier settings. While EnvModel competitive in identifying nighttime outliers, it is inferior to the other methods against the other considered outliers. These results suggest that proposed method effectively detects outliers for realistic problems, such autonomous driving.
8 Discussions and Conclusion
In this paper, we present the first detailed study of distance-based anomaly detection framework in deep RL, considering random, adversarial, and OOD state outliers in both offline and online settings. The primary detection backbone is based on Mahalanobis distance, and we extend it to robust and distribution- free versions by leveraging robust estimation and conformal prediction techniques. Experiments on classical control environments, Atari games, and the autonomous driving environment, demonstrate the effectiveness of our proposed methods in detecting the three types of outliers. The conformal MD method achieves the best detection performance in most scenarios, especially in the online setting. Our research contributes to developing safe and trustworthy RT systems real-world applications.
Limitations and Future Work. In the online setting, especially with high proportion of outliers, it may be preferable to denoise the detected state outliers via some neighboring smoothing techniques, e.g. mirup (Zhang et al. 2018 Wang et al. 2020), rather than deleting them directly as performed in this paper. To relax the Gaussian assumption in the hypothesis test of our detection, we can consider other non-parametric methods, such as one-class support vector machines [Choi] 2009) or isolation forests {Liu et al. 2008). substantial challenge that remains for future work is to devise a more informed detector to distinguish between real "bad" outliers that can cause truly mislcading actions and "good" new sam- ples collected through exploration, which can potentially benefit the policy learning, especially for image inputs (Zhang & Ranganath; 2023
16

---

Published in Transactions on Machine Learning Research (10/2024)
Acknowledgements
Ke Sun was supported by the State Scholarship Fund from China Scholarship Council (N0:202006010082) Linglong Kong was partially supported by grants from the Canada CIFAR AT Chairs program, the Alberta Machine Intelligence Institute (AMII), and Natural Sciences and Engineering Council of Canada (NSERC), and the Canada Rescarch Chair program from NSERC. Hongming Zhang and Martin Müller were supported by UAHJIC, the Natural Sciences and Engineering Research Council of Canada (NSERC), and an Amii Canada CIFAR AI Chair.
References
David Abel, André Barreto, Benjamin Van Roy, Doina Precup, Hado P van Hasselt, and Satinder Singh. A definition of continual reinforcement learning. Advances in Newral Information Processing Systems, 36, 2024.
Amurag Ajay, Abhishek Gupta, Dibya Ghosh, Sergey Levine, and Pulkit Agrawal. Distributionally adaptive meta reinforcement learning. Advances in Newral Information Processing Systems, 35:25856 15:25856-25869, 2022.
Nishanth Anand and Doina Precup. Prediction and control in continual reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Anastasios Nikolas Angelopoulos, Stephen Bates, Michael Jordan, and Jitendra Malik. Uncertainty sets for image classifiers using conformal prediction. In International Conference on Learning Representations, 2021.
Fengshuo Bai, Hongming Zhang, Tianyang Tao, Zhiheng Wu, Yanna Wang, and Bo Xu. Picor: Multi-lask deep reinforcement learning with policy correction. Proceedings of the AAAI Conference on Artificial Intelligence, 37(6))6728 6736, Jun. 2023. doi: 10.1609/ va..//aa.........22222 v3766.2582.. 25825. URL https://ojs. www.s.aaaa...oom org/ index. index.php/ http:ppppw../aarrrreeee/////////
Fengshuo Bai, Runze Liu, Yali Du, Ying Wen, and Yaodong Yang. BATTLE: Towards behavior-oriented adversarial attacks against deep reinforcement learning, 2024a. URL https://pperrrrieernntt/oo/ id=rp5vfyp5Np
Fengshuo Bai, Rui Zhao, Hongming Zhang, Sijia Cui, Ying Wen, Yaodong Yang, Bo Xu, and Lei Han. Efficient. sifference-based reinforcement learning via aligned experience estimation. arXiv preprinl arXiv:2405.18688 2024b.
Rina Foygel Barber, Emmanuel Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics, 51(2):816 845, 2023.
Andrew G Barto, Richard S Sutton, and Charles W Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions systems, man, and cybernctics, 5:834-846, 1983.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The Arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253 279, June 2013.
Justin Boyan and Andrew Moore. Generalization reinforcement learning: Safely approximating the value function. Advonces in newral information processing systems, 1994.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAI Gym, 2016. URL httpp////aaaav..r///a/////////// 01540
01540
RW Butler, PL Davies, and M Jhun. Asymptotics for the minimum covariance determinant estimator. The Annals of Statistics, pp. 1485-1400, 1993.
Yuanjiang Cao, Xiaocong Chen, Lina Yao, Xianzhi Wang, and Wei Emma Zhang. Adversarial attacks and detoction on reinforcement learning- interactive recommender systems. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1672, 2020.
17

---

Published in Transactions on Machine Learning Research (10/2024)
Stephanie C.Y. Chan, Samuel Fishman, Anoop Korattikara, John Canny, and Sergio Guadarrama, Mcasuring the reliability of reinforcement learning algorithms. In International Conference Learning Represenla- tions, 2020.
Shang-Tse Chen, Cory Cornelius, Jason Martin, and Duen Horng Chan. ShapeShifter: Robust physical adversarial attack on faster R-CNN object detector. In Machine Learning and Knowledge Discovery in Dalabases: Europeun Conference, ECML PKDD 2018, Dublin, Irelond, Seplember 10 14, 2018, Proceed- ings, Part 1 18, pp. 52-62 68. Springer, 2019.
Young-Sik Choi. Least squares one-class support vector machine. Pallern Recognalion Letters, 30(13):1236- 1240, 2009.
Mohamad H Danesh and Alan Fern. Out-of-distrbbution dynamics detection: RL-relevant benchmarks and results. ICML 2021 Workshop on Uncertainty and Robustness in Deep Learning. 2021.
Roy De Macsschalck, Delphine Jouan-Rimbaud, and Désiré L Massart. The Mahalanobis distance. Chemo- metrics and Intelligent Laboralory Systems, 50(1):1 18, 2000.
Hao Dong, Zihan Ding, and Shanghang Zhang. Deep Reinforcement Learning: Fundamentals, Research and Applications. Springer Nature, 2020.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Conference on Robot Learning, pp. 1-16. PMLR, 2017.
Nebrase Elmrabit, Feixiang Zhou, Fengyin Li, and Huiyu Zhou. Evaluation of machine learning algorithms for anomaly detection. In 2020 international conference on cyber security and protection of digital services (cyber security), pp. 1 8. IEEE, 2020.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Javier Gareia and Fernando Fernández. A comprehensive survey on safe reinforcement learning, Journal of Machine Learning Research, 16(1))1437-1490, 2015.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farloy, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014a.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. International Conference on Learning Representations (ICLR), 2014b.
R. Grübel. A minimal characterization of the covariance matrix. Metrika, 35(1):49 52, 1988.
Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll. review of safe reinforcement learning: Methods, thoory and applications, 2024. URL https: orw..bs///
2205. 10330
Tuomas Haarnoja, Aurick Zhou, Picter Abbecl, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Tom Haider, Karsten Roscher, Felippe Schmoeller da Roza, and Stephan Günnemann, Out-of-distributt detection for reinforcement learning agents with probabilistic dynamics models. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, pp. 851-859, 2023.
Johanna Hardin and David M Rocke. The distribution of robust distances. Journal of Computational and Graphical Statistics, 14(4):928-9465 2005.
Trevor Hastie, Robert Tibshirani, Jerome Friedman, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
18

---

Published in Transactions on Machine Learning Research (10/2024)
Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety. arXiv preprint. ar Xrxin:2199 13916, 2021.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference on artificial intelligence, 2018.
Walced Hilal, S Andrew Gadsden, and John Yawney. Financial fraud: a review of anomaly detection techniques and recent advances. Expert systems With applications, 193:116429, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6855 2020.
Jincheng Hu, Jihao Li, Zhuoran Hou, Jingjing Jiang, Cunjia Liu, Liang Chu, Yanjun Huang, and Yuanjian
Zhang. Potential auto-driving threat: Universal rain-removal attack. Iscience, 26(9), 2023.
Liyuan Hu, Mengbing Li, Chengchun Shi, Zhenke Wu, and Piotr Fryzlewicz. Doubly inhomogeneons rein- forcement learning, 2022. URL httpps:/arxiv..rrgg abwgg/bb//aa 221. 1.99883
Sandy Huang, Nicolas Papernot, Tan Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks neural network policies. International Conference Learning Representations (ICLR) workshop, 2017.
Peter J Huber. Robust statistics, volume 523. John Wiley Sons, 2004.
Mia Hubert and Michiel Debruyne. Minimum covariance determinant. Wiley interdisciplinary reviews: Compulational stutislics, 2(1):36 43, 2010.
Vladislav Ishimtsev, Alexander Bernstein, Evgeny Burnaev, and Ivan Nazarov. Conformal k-nn anomaly detector for univariate data streams. In Conformal and Probabilistic Prediction and Applications, pp. 213-227. PMLR, 2017.
Scott Jordan, Yash Chandak, Daniel Cohen, Mengxue Zhang, and Philip Thomas. Evaluating the per- formance reinforcement learning algorithms. In International Conference on Machine Learning, pp. 4962 4973. PMLR, 2020.
Ryo Kamoi and Kei Kobayashi. Why is the mahalanobis distance effective for anomaly detection?, 2020. URL
Kiarash Kazari, Ezzeldin Shereen, and György Dán. Decentralized anomaly detection in cooperative multi- agent reinforcement learning. In IJCAT, pp. 162-170, 2023.
Khimya Khetarpal, Matthew Riemer, Trina Rish, and Doina Precup. Towards continual reinforcement learn- ing: A review and perspectives. Journal of Artificial Intelligence Research, 75:1401-1476, 2022.
B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey, IEEE Transactions on Intelligent Transportation Systems, 23(6):4909-4926 2021.
William Klecka, Gudmund R Iversen, and William Klecka. Discriminant analysis, volume 19. Sage, 1980.
Rikard Laxhammar and Göran Falkman. Sequential conformal anomaly detection in trajectories based on Hausdorff distance. In 14th international conference on information fusion, pp. 1-8, IEEE, 2011.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of- distribution samples and adversarial attacks. Advances in Neural Information Processing Systems, 31, 2018.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020. URL http:://arrii.oor//a///000.....
19

---

Published in Transactions on Machine Learning Research (10/2024)
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Contimuous control with deep reinforcement, learning. International Conference on Learning Representations (ICLR), 2016.
Fei Tony Liu, Kai Ming Ting, and Zui-Hua. Zhou. Isolation forest. In 2008 Eighth. IEFE International Conference Data Mining, pp. 413 422. IEEE, 2008.
Runze Liu, Fengshuo Bai, Yali Du, and Yaodong Yang. Meta-reward-nett Implicitly differentiable reward learning for preference-based reinforcement learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advonces in Newral Information Processing Systems, volume 35, pp. 22270-
22284. Curran Associates, Inc., 2022. URL https::/pproceeddngg http:///ceeddnnn..eesrrs..s..o wwwripp.cc//ppess.ee. https//eeppeeeeeeeeeepeppeeeee files/paper/ http:///ww.ccaaaaoo......//////r httpppps//pppreiigg..ee..s.....s
pdf
Ricardo A Maronna and Victor J Yohai. Robust estimation of multivariate location and scatter. Wiley StatsReff Statistics Reference Online, pp. 1 12, 2014.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529 533, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Inter- national conference on machine learning, pp. 1928 1937. PMLR, 2016.
Andrew William Moore. Efficient casorr-based learning for robot control. Technical report, University of Cambridge, 1990.
Robert Müller, Steffen Illiu,, Thomy Phan, Tom Haider, and Claudia Linnhoff-Popien Towards anomaly detection in reinforcement learning. In Proceedings of the 21st International Conference on Aulonomous Agents and Multiagent Systems, pp. 1799-1803, 2022.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta-reinforcemen learning. arXiv preprint Xiv::18833 1347, 2018.
Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: A review. ACM Computing Surveys (CSUR), 54(2):1-38, 2021.
Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. Inductive confidence ma- chines for regression. In Machine learning: ECML 2002: 13th European conference on machine learning Helsinki, Finland, August 19-23 2002 proceedings 13, pp. 345-356 Springer, 2002.
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimice Deep multitask and transfer reinforcement learning. arXiv preprint arXi::51110063/2 1.06342, 2015.
Gaurav Parmar, Taesung Park, Srinivasa Narasimhan, and Jun Yan Zhu, One-step image translation with text-to-image models, 2024. URI. htttps://arxiv.org//aaaa http//3ab//224..../// 2403. 12036
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforeement learning with adversarial attacks. Advances in Neural Information Processing Systems, 2017.
Andrew Patterson, Samuel Neumann, Martha White, and Adam White. Draft: Empirical design in rein- forcement lcarning. Journal of Artificial Intelligence Research, 1, 2020.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. International Conference on Machine Learning (ICML), 2017.
Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padby, and Balaji Lakshminarayanan. A simple fix to Mahalanobis distance for improving near-OOD detection. International Conference on Machine Learning (ICML) workshop, 2021.
20

---

Published in Transactions on Machine Learning Research (10/2024)
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' dlAcc-BBuu,, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Peter J Roussceuw. Least median of squares regression. Journal of the American Statisticnl Association, 79 (388):871 880, 1984.
Peter J Roussceuw and Bert C Van Zomeren. Unmasking multivariate outliers and leverage points. Journal of the American Statistical Association, 85(411):633 639, 1990.
John Schulman, Sergey Levine, Picter Abbecl, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of The 32nd International Conference on Machine Learning, pp. 1889 1897.
PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy oplimiza- tion algorithms, 2017. URL https::/arrt http::///arxii..orr/////////1// wwwpw///w///../oo/////..
06347
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional con- tinuous control using generalized advantage estimation. International Conference on Learning Represen- tations, 2018.
Andreas Sedlmeier, Thomas Gabor, Thomy Phan, Lenz Belzner, and Claudia Linnhoff-Popien Uncertainty- based -ntto--distribution classification in deep reinforcement learning. In Proceedings of the 12th In- ternational Conference on Agents and Artificial Intelligence. SCITEPRESS Science and Technology Publications, 2020a. doi: 10.5220/00088 0090000000999992252...
Andreas Sedlmeier, Robert Müller, Steffen Miimm, and Claudia Linnhoff-Popien. Policy entropy for out- of-distribution classification. In Artificial Neural Networks and Machine Learning ICANN 2020: 29th International Conference on Artificial Neural Networks, Bratislava, Slovakio, Seplember 15 18, 2020, Proceedings, Part II 29, pp. 220-431.21 Springer, 2020b.
Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research. 9(3), 2008
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. International Conference on Leorning Representations (ICLR), 2013.
Matthew Taylor and Peter Stone. Transfer learning for reinforcement learning domains: survey. Journal of Machine Learning Research, 10(7), 2009.
Jiaye Teng, Chuan Wen, Dinghuai Zhang, Yoshua Bengio, Yang Gao, and Yang Yuan. Predictive inference with feature conformal prediction. In The Eleventh International Conference on Learning Representations, 2023.
Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal prediction under covariate shift. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (ods.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 TEEE/RS) International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012. doi: 10.1109/IROS. R0..199/RRS.202..000 200.000122800 6386109.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Vladimir Vork, Alexander Gammerman, and Glenn Shafer. Algorithmic learning a random world, vol- ume 29. Springer, 2005.
21

---

Published in Transactions on Machine Learning Research (10/2024)
Chen Wang, Sarah Erfani, Tansu Alpcan, and Christopher Leckie. Oil-ad: An anomaly detection framework for sequential decision sequences, 2024. URL
04567
Kaixin Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. Improving generalization in reinforcement learning with mixture regularization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.); Aduances in Neural Information Processing Systems, volume 33, pp. 7968-7978. Curran Associates, Inc., 2020.
Edwin Wilson and Margaret M Hilferty. The distribution of chi-square. Proceedings of the National Academy of Sciences of the United States of America, 17(12):684, 1931.
Wei Xiao, Xiaolin Huang, Fan He, Jorge Silva, Saba Emrani, and Arin Chaudhuri. Online robust principal component analysis with change point detection. IEEE Transactions on Multimedia, 22(1):59 68, 2019.
Zhongwen Xu, Hado P van Hasselt, and David Silver. Mela-gradient reinforcement learning. Aduances in neural information processing systems, 31, 2018.
Hongming Zhang and Tianyang Yu. Taxonomy of reinforcement learning algorithms. Deep reinforcement learning: Fundomentals, research and applications, pp. 125-133, 2020.
Hongming Zhang, Chenjun Xiao, Han Wang, Jun Jin, Xu Bo, and Martin Müller. Replay memory as an empirical mdp: Combining conservative estimation with experience replay. The Eleventh International Conference on Learning Representations, 2023a.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Duane Boning, and Cho-Jui Hsich. Robust deep rein- forcement learning against adversarial perturbations on observations. Advances in Neural Information Processing Systems, 2020.
Lily H Zhang and Rajesh Ranganath. Robustness to spurious correlations improves semantic out-of- distribution detection. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 15305-15312, 2023.
Yingying Zhang, Chengehun Shi, and Shikai Luo. Conformal off-policy prediction. In International Confer- ence Artificial Intelligence and Statistios, pp. 2751-2768. PMT.R, 2023b.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference on computer vision, pp. 2223-2232 2017.
Zhuangdi Zhu, Kaixiang Lin, Anil K Jain, and Jiay Zhou Transfer learning in deep reinforcement learning: A survey. IEEE Transactions on Pollern Analysis and Machine Intelligence, 2023.
22

---

Published in Transactions on Machine Learning Research (10/2024)
A Proof of Proposition
Proof. We show that for cach action class c, the square of Mahalanobis distance d is identically independent Chi-squared distributed under the Gaussian assumption. Without loss of generality, we denote and Σ as the mean and variance matrix of the closest class-conditional Gaussian distribution. We need to show = (f(s) - p)) 2 (f(ss) - ()) Chi-squared distributed. Firstly, by eigenvalue decomposition, we have
Σ
u
(9)
where 2e and up are the k-th eigenvalue and eigenvector of E. Plugging it into the form of d, we immediately obtain
d d=(((s) - (1) 1 p)
-(f(s) - -M) 2 No uu p))
(f((s) 14) un (J(s) p))
(10)
Nx u (J(s) mm]
X
where X is Gaussian variable that results from the linear transform of Gaussian distribution f()) where f(s) ~ NN,, E). Therefore, the resulting variance of can be derived as
t xx 2 uuf Ed uu -x=uu Uu 2 Auuuu Juu
Σ N Ayu uuu 1=1
(11)
As the μl and μt are orthogonal if j o k, the variance of can be further reduced to
t Xp Au ug un = 1|uu |ux| 2=
(12)
Each X is a standard Gaussian distribution. Then we have d, the square of Mahalanobis distance, Chi- squared distributed, i.e., (p), independent the action class c. Without loss of generality, the smallest d among all action classes, i..,, (s), also a Chi-squared distribution. That is say, M(s) ~ x²()).
B Results in Offline Setting
B.1 Results across Different Noise Strengths
We provide detailed detection accuracy of various detection methods across different noise strengths. The results two classical control environments and six Atari games are gived in Tables 5 and 6
B.2 Visualization of Outlier States on Six Games
We plot the outlier states on Breakout, Asterix, and Spacelnvaders games in Fig. 7 and outliers states on Enduro, FishingDerby, and Tutankham in Fig.
23

---

Published in Transactions on Machine Learning Research (10/2024)
Table 5: Detection accuracy (%) of our MD, Robust MD, and conformal MD strategies compared with other
baseline methods two classical control environments with = 0.05.
Environments
Outliers
Perturbation
ED
TMD
PEOC
EnvModel
MD
RMD
MD ID
std=0.3
60.54
93.24
50.00
50.01
94.74
78.7
95.33
Random
std=0.5
75.54
94.49
50.00
49.98
96.14
79.15
97.00
Cartpole
c=0.15
50.58
92.53
50.00
49.98
93.65
78.48
93.79
Adversarial
c=0,2
51.65
93.89
50.00
50.01
95.31
78.97
95.76
0OD
MountainCar
87.27
94.26
93.84
96.45
79.09
97.46
std=0.3
86.91
85.82
50.00
50.11
89.81
77.84
92.52
Random
std=0.5
92.08
86.96
50.00
19.71
91.37
78.35
94.73
MountainCar
c=0.001
63.86
80.72
50.00
19.69
85.09
74.04
86.83
Adversarial
c=0.01
64.13
81.30
50.00
49.693
85.72
74.63
87.32
OOD
Cartpole
90.89
86.51
50.00
48.73
90.49
77.21
91.69
Table Detection accuracy (%) our MD, Robust MD, and conformal MD strategies compared with other
baseline methods on Atari games with a 0.05.
Games
Outliers
Perturbation
ED
TMD
PEOC
EnvModel
MD
RMD
MD+C
std=0.02
50.13
52.01
50.00
50.00
54.89
62.80
54.46
Random
std=0.04
56.18
66.26
50.00
50.00
67.85
79.64
66.76
6=0.001
81.28
87.44
50.00
50.00
89.39
79.85
89.54
Breakout
Adversarial
6=0.11
87.36
90.67
50.00
50.00
92.23
80.57
93.87
Asterix
66.80
47.91
49.99
97.50
50.32
80.73
51.15
OOD
SpaceInvaders
47.07
47.74
49.99
97.50
48.92
78.22
50.37
std=0.1
42.56
42.87
50.01
49.99
48.28
63.05
49.04
Random
std=0.2
45.29
47.37
50.01
49.99
72.31
75.86
60.31
==0.001
83.41
85.31
50.01
50.00
91.38
75.02
93.62
Asterix
Adversarial
6=0.01
83.94
85.90
50.01
50.00
91.95
75.40
94.35
Breakout
41.73
42.94
50.01
48.95
48.24
75.72
51.85
OOD
SpaceInvaders
37.38
38.61
50.01
58.55
43.92
39.62
47.50
std=0.02
50.00
48.46
50.01
50.00
53.26
75.24
52.53
Random
std=0.0M
52.88
79.38
50.01
50.00
87 .2.0
83.57
81.62
==0.001
67.13
89.69
50.01
50.00
95.64
83.16
95.87
SpaceInvaders
Adversarial
6-0.1
74.66
90.93
50.01
50.00
96.65
79.07
97.05
Breakout
45.81
45.64
50.01
48.97
56.56
78.88
54.23
OOD
Asterix
44.71
46.22
50.01
48.72
57.45
83.06
53.03
std=0.1
49.42
45.27
50.00
50.00
54.03
81.65
52.24
Random
std=0.2
48.67
72.70
50.00
49.99
91.40
83.15
88.17
==0.001
91.86
91.26
50.00
49.99
96.24
83.27
97.48
Enduro
Adversarial
<=0.01
93.93
91.26
50.00
49.99
96.24
83.37
97.48
FishingDerby
63.95
83.15
50.00
47.54
85.10
83.44
61.39
OOD
Tutankham
50.19
66.60
50.00
47.67
74.98
83.28
65.13
s.2
48.80
48.20
50.00
50.00
51.65
83.77
50.82
Random
std=0.3
49.03
84.60
50.00
50.00
87.86
87.71
82.09
e00.001
83.44
92.48
50.00
50.01
97.31
86.90
97.33
FishingDerby
Adversarial
0.0.
89.12
92.54
50.00
50.01
97.49
87.41
97.46
Enduro
48.89
56.69
50.00
75.34
60.31
86.45
59.78
OOD
Tulankham
53.64
55.65
49.96
47.97
57.93
76.59
57.25
std=0.04
50.00
48.30
50.00
50.03
49.31
71.49
50.00
Random
std=0.06
50.00
46.68
50.03
50.00
48.79
76.51
49.57
<00.1
60.08
89.37
50.03
50.01
95.24
77.07
95.56
Tutankham
Adversarial
c0..5
72.36
89.55
50.03
50.01
95.28
77.05
97.49
Enduro
49.96
89.17
50.03
97.50
95.12
77.18
91.59
OOD
FishingDerby
50.0
77.34
50.03
97.50
84.12
77.17
67.77
24

---

Published in Transactions on Machine Learning Research (10/2024)
003
003
DOOO DOOO
Breakout: Clean
(b) Breakout: Random
(c) Breakout: Adversar-
(d) Breakout: Asterix
(e) Breakout: Space
Asterix: Clean
(g) Asterix: Random
(h) Asterix: Adversarial
(i) Asterix: Breakout
ii) Asterix: Space
DOO
R
Spaces Clean
(1) Space: Random
(m) Space: Adversarial
(n) Space: Breakout
(0) Space: Asterix
Figure 7: Visualization of various state outliers on Breakout, Asterix, and Spacelnvaders games.
25

---

Published in Transactions on Machine Learning Research (10/2024)
00411
0941L
(a) Enduro: Clean
(b) Enduro: Random
(c) Enduro: Adversarial
(d) Enduro: Fishing
(e) Enduro:
Tu-
tankham
Fishing: Clean
(g) Fishing: Random
(h) Fishing: Adversar-
(i) Fishing: Enduro
(i) Fishing: Tutankham
(k) Tutankham: Clean (1) Tutankham:
Ran-
(m) Tutankham: Adv
(n)
Tutankham:
En- (o) Tutankham:
Fish-
dom
duro
ing
Figure 8: Visualization of various state outliers Enduro, Fishing and Tutankham games.
26

---

Published in Transactions on Machine Learning Research (10/2024)
B.3 Effectiveness of Robust MD
We take the cubic root of the Mahalanobis distances, yielding approximately normal distributions (Wilson Hilferty 1931). In this experiment, 250 clean states are drawn from the replay buffer, and 50 abnormal states are drawn from each of the three types of outliers. We reduce the state feature dimension to 2 via t-SNE and compute Mahalanobis distances of these two kinds of states to their centrality within each action class under the estimation based on MD or Robust MD, respectively. Fig. g suggests that Robust MD separates inliers and outliers better than MD on Breakout within a random action class, indicating its effectiveness in detecting RL evaluation. Similar results are also given in other games.
We plot the distributions of inliers and three types of outliers on SpaceInvaders and Asterix games in Figs. and [1 respectively. It is worth noting that Robust MD also capable of enlarging the separation of distributions between inliers and both random and adversarial outliers Spacelnvaders game, while its benefit seems to be negligible on OOD outliers (Breakout) on SpaceInvaders games as well as in Asterix game. We speculate that it is determined by the game's difficulty. Specifically, the PPO algorithm can
Using non-rabust estimates (MD)
Using robust estimates (Robust MD)
Using non-robust estimates (MD)
Using robust estimatas (Robust MD)
Using non-rabust astimates (MD)
Using robust estimates (Robust MD)
ViMahaal dit
200
Inliers
Inliers
Inliers
Adversarial
00D
00D
(a)
Random Outliers.
(b) Adversarial Outliers.
(c) OOD Outliers.
Figure 9: Boxplot of distributions between inliers and three types of outliers in an action class on Breakout game.
Using non-rebust estimates Using robust estimates MD)
Using non-robust estimates (MD)
Using robust estimaates (Robust MD)
Using non-robust estimates (MD)
Using robust estimates
(MD)
(PPennal 35t.
M'Mahaa des..
(Meiaal dist.)
Intlers
Random
Inliers
Random
Inliers
Adversarial
Iniiers
Adversarial
Intlers
00D
Inllerrs
000
(a) Random Outliers.
(b) Adversarial Outliers.
(c) OOD Outliers.
Figure 10: Boxplot of distributions between inliers and three lypes of outliers in an action class on Spaceln- vaders game.
Using non-rebust estimates (MD)
Using robust estimates (Robust MD)
Using non-rabust astimates (MD)
Using rabust estimates (Robust MD)
Using non-rabus estimates (MD)
Using robust estimates
(Robust MD)
(iKohall disk.
Inliers
Random
Inliers
Random
Inliers
Adversarial
Inilers
Adversarial
Inllers
OOD
Inliers
OOD
(a) Random Outliers.
(b) Adversarial Outliers.
(c) OOD Outliers.
Figure 11: Boxplot of distributions between inliers and three types of outliers in an action class on Asterix
game.
27

---

Published in Transactions on Machine Learning Research (10/2024)
achieve desirable performance on the simple Breakout game, thus yielding informative feature space vectors. By contrast, there is room for the generalization of PPO on both SpaceInvaders and Asterix games, such that Robust MD might not help when handling the less meaningful state feature vectors in these two games.
B.4 Sensitivity Analysis
We provide the sensitivity analysis of Robust MD in terms of the PCA dimension in Fig. 12 of the number of principal components on the detection performance for robust MD detection is shown in Fig. 12 The detection accuracy over all considered outliers improves as the number of principal components increases, except for a slight decline for random and adversarial outliers (red and blue lines) on the Breakout game. The increase implics that the subspace spanned by principal components with small explained variance also contains valuable information for detecting anomalous states from in-distribution states, which coincides with the conclusion in (Kamoi & Kobayashi 2020)
The impact
The result of MD estimation manifests in Fig. 13 It suggests that there is still an ascending lendency of detection accuracy as the number of principal components increases.
Breakout
Asterix
SpaceInvaders
Detection Accuracy(%6
Detection Accuracy(%)
Detection Accuracy('
Random
Adversorial
OOD
Rundom Adversarial OOD
Random
Adversrrial
70-
OOD
50
103
150
200
50
100
150
200
50
100
150
200
Principal Components
Principal Components
Principal Components
Figure 12: Detection performance under Robust MD as the number of principal components increases.
Breakout
Asterix
SpaceInvaders
Detection Accuracy(%% (%)
Detection Accuracy(%o)
Detection Accuracy
10-
Randorn 00D
Random
Random
60
60
Adversarial
60
Advensarial
OOD
oop
50
50
160
150 200
150
Principal Components
Principal Components
Principal Components
Figure 13: Detection performance under MD as the number of principal components increases.
28

---

Published in Transactions on Machine Learning Research (10/2024)
C Results in Online Setting
C.1 Setup and Full Main Results
As supplement to the results on the main pages, we provide the whole results on two feature-input tasks and six Atari games from Fig. 14 to Fig. 21 The 'Mean Score" in the first row indicates the accumulated rewards of PPO, and the 'F1 Score" the second row shows the detection performance during RL training, The F1 score computed based on precision and recall. The third row shows the relationship between the average F1 score and policy performance during training. We can find that higher detection accuracy is generally associated with better policy performance. We also find that the cumulative reward is not strongly correlated with detection ability in some games. A high detection accuracy may only improve the cumulative reward to small degree. This suggests that we need more metrics to mcasure the effect of our detection performance more ivel. Hyperparameters in our mods are shown in Table
Hyperparameter
Value
Confidence level (1-a)
1-0.05
Moving window size (m)
5120
Sample size (Ne)
2560
Iteration (K)
10000 (1e7 steps in total)
Environment number (N)
8
Horizon (T)
128
Table 7: Hyper-parameters in the training phase. RI-related parameters are the same as those of the PPO algorithm.
29

---

Published in Transactions on Machine Learning Research (10/2024)
600
400
Mean Sagre
200
Heen Scor
200
Neen Soute
300
385
100
Step
Step
Step
Stea
2.0
1.0
Scor
Step
Stap
Skep
ED MD+C
JE0
460
MD+c
ND+C
BMD
PEOC
EnvMede
Avurage Putter
JEEC
Enu-Mdee
322
PEoC
RVD
MD
Ste
4D
JMD
238
Random
Ro mdam
Random
Random
D7 23
1b
009
1.3
36
0X
10
Score
(a) Gaussian std=1
(b) Gaussian std=0.3
OOD MountainCar
(d) Adversarial
Figure 14: Detection performance across various state outliers in the online training CartPole.
30

---

Published in Transactions on Machine Learning Research (10/2024)
-120
-12
Welt St0V
-148
Me4e Score
Meam Scom
10
240
182
-10
220
-20
-231
.20
Strp
210
29-
0.0-
0.0
11.5.00
11.50
0.2
0.
Stap
Stap
Step
Step
-240
Auto
-152
"ND-C
-240
1350
MD
ChuMode
Arerage Performance
Loc
JND
Aoar Puffrmance
-2.0
JDD
2200
ND=C
ErwMtdd
.ND
PEo:
-10
X/0
Je0c
(a) Gaussian std=1
(b) Gaussian std=0.3
(c) OOD Cart
(d) Adversarial
Figure 15: Detection performance across various state outliers the online training on MountainCar.
31

---

Published in Transactions on Machine Learning Research (10/2024)
2951
206
Recue
235
225
208-
375-
359
Maan
125-
300-
15
50-
Score
P9
0.0
Frintt
Diveatio
Beration
Beeetion
Pertiion
200
220
227
226
200
Fea porformaee seee
PMC
J90
20
320
JED
Bandom
Bandom
Rancan
100
Randon
30
Score
Score
Score
F1.cam
F1.core
(a) Gaussian std=1
(b) Gaussian std=0.3
(c) OOD SpaceInvaders
(d) OOD Asterix
(e) Adversarial
Figure 16: Detection performance across various state outliers in the online training on Breakout.
32

---

Published in Transactions on Machine Learning Research (10/2024)
2000
2003
2000
2000
2253
2000
140
2480
1443
Score
bo
6.0
6.0
6.0
0.0
Trint
Diveatin
Beration
Beretion
Pention
2099
Adbo
2000
2830
243
2000
210
2330
JHD
150
JMD
2300
AND
-150
205)
JND
22000
Rardem
2812
253
Sandon
Jandam
10
Bandam
200
(a) Gaussion std=1
(b) Gaussian std=0.3
(c) OOD Breakout
(d) OOD Spacelnvaders
(e) Adversarial
Figure 17: Detection performance across various state outliers in the online training on Asterix.
33

---

Published in Transactions on Machine Learning Research (10/2024)
C58
110
420
300
0.
Scare
69
0.0
Frint
Diveatio
Beration
Beestion
Pention
2o Auci HD4r
M010
J90
20
Auza
Auto
Aute
MD
Fin Pertemparce
TVD
Faa Pesfomuacce
kolc
200
VDV
Fb
Bandom
TMD
AND
F3
300
38D
330
17
37
14
13
13
33
22
500
12
27
33
00
12
02
35
Gaussian std=1
(b) Gaussian std=0.3
(c) OOD Breakout
(d) OOD Asterix
(e) Adversarial
Figure 18: Detection performance across various state outliers in the online training SpaceInvaders.
34

---

Published in Transactions on Machine Learning Research (10/2024)
159
J53-
240
200-
23.
6.0
6.0
6.
0.3-
Scare
140
140
Erint
Itrestion
Remttoon
Fention
MM
1900
MB+C
Arr
JUD
Aul
371
Fentemaaee
JTD
310
Randam
JHQ
12
32
38
24
30
13
05
12
20
33
04
33
65
12
35
14
Gaussian std=1
(b) Gaussian std=0.3
(c) OOD Tutankham
(d) OOD FishingDerby
(e) Adversarial
Figure 19: Detection performance across various state outliers in the online training on Enduro.
35

---

Published in Transactions on Machine Learning Research (10/2024)
794
Mean Soom
-78
-20
425
6.0
0.0
6.0
0.3-
Seore
140
140
Frint
Rention
Fention
-74
JMD
xp-r
ANDALL
74
JHD
76
3
Rost Peftmmate
-70
Fin Notemmaaee
AMD
Aute
JMD
-77
Augo
TNb
3
JMD
20
%
Jancom
Randem
Random
JND
-2
Score
Pl Slore
500%
Score
51
(a) Gaussian std=1
(b) Gaussian std=0.3
(c) OOD Enduro
(d) OOD Tutankham
Adversarial
Figure 20: Detection performance across various state outliers in the online training on FishingDerby.
36

---

Published in Transactions on Machine Learning Research (10/2024)
190-
166
369
300
345
6.0
6.0
6.0
0.3-
E.C.
044
140
140
Erinnt
Itreation
Beestion
Rention
Fention
1.00
EnsModal
EnrW aal
FEO: Auto
Erieedde
U05
Ensttdal Mbac
ALl
175
ND-C
reffemant
AMO
170
170
THIT
Mn
TD
JND
JMD
Random
Rantom
397
PMD
PEDC
150
17
26
30
13
120
05
22
12
32
22 02
12
14
Gaussian std=1
(b) Gaussian std=0.3
(c) OOD Enduro
(d) OOD FishingDerby
(e) Adversarial
Figure 21: Detection performance across various state outliers in the online training Tutankham.
37

---

Published in Transactions on Machine Learning Research (10/2024)
C.2
Ablation Study Double Anomaly Detectors
Fig. 22 reveals that double self-supervised delectors help adjust the detection errors and improve the detection accuracy compared with the single detector. MD with double detectors outperforms MD with a single detector significantly, although RMD with double detectors comparable to RMD with a single detector.
1.0
accuracy
accuracy
0.6
0.6
Robust MD Double
Robust MD Double
MD Double
MD Double
Robust MD Single
Robust MD Single
MD Single
MD Single
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
iteration
iteration
(a) Gaussian std=1.
OOD Asterix.
Figure 22: The detection accuracy with and without double self-supervised detectors on Breakout with random and OOD outliers on Breakout.
C.3 Ablation Study on Number of Noisy Environments
We train PPO in two, four, or six noisy environments with random and OOD outliers among all eight parallel environments. We use PCA to reduce the feature vectors to 50 dimensions and estimate the detector using Robust MD. Fig. 23 illustrates that compared with the Auto baseline, our RMD method is robust. when encountering different ratios of outliers, especially with higher contamination ratio. The dashed lines in different colors represent Auto basclines that correspond to the different number of noisy environments. The training performance with our detection method gradually approaches the ideal baselines, i.e., Auto.
300
6
300
6
250
250
mean score
2
mean score
2
200
200
100
100
50
50
10000
iteration
iteration
(a) Gaussian std=0.3
(b) OOD Asterix.
Figure 23: Training performance under Robust MD detection under different proportions outlier exposure on Breakout. (2, 4, out of environments).
38


Reviews:
[{"review_id": "mcTVYXTYQg", "reviewer": "Reviewer_7cuR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The writing is clear to follow and understand.\n2. The authors perform comprehensive evaluations.\n3. The empirical performance looks good.\n4. The motivation parts are substantial and of high quality.\n\nWeakness:\n1. Discretizing policy to several classes might not be practical in some cases and suffers from the curse of dimensionality.\n2. The computational cost is too heavy, maintaining the $\\mu$ and $\\Sigma$ for each class requires much more data.\n3. The paper only evaluates image-input tasks, where there is a possibility that the shape of sampled distributions is nicer and more centered.\n4. There is no guarantee that the method will work for those state-action pairs with low visit probability. There might not be enough samples to give a reasonable identifier.\n5, As the authors addressed in the limitation, the algorithms might regard the new exploration activities as outliers.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper \"A Distance-based Anomaly Detection Framework for Deep Reinforcement Learning\" presents a novel framework called MDX for anomaly detection in deep reinforcement learning (RL) systems. Its main technical contributions include:\n\n**Mahalanobis Distance-based Detection**: The framework uses the Mahalanobis Distance (MD) within a hypothesis testing framework under Gaussian assumptions to identify random, adversarial, and out-of-distribution (OOD) state outliers. The MD-based detection is extended with a robust MD version using Minimum Covariance Determinant (MCD) to handle noisy data more effectively. The authors introduce a non-parametric conformal prediction method to address limitations of the Gaussian assumption, which allows for a distribution-free anomaly detection.\n\n**Offline and Online Settings**: The framework supports both offline (fixed datasets) and online (dynamic datasets with continuous interaction) RL settings, making it flexible for real-world applications. In the online RL setting, the paper introduces a technique that maintains two self-supervised detectors—one for inliers and one for outliers—to improve detection performance as the policy evolves during training.\n\n**Comprehensive Evaluation**: Extensive experiments are conducted in both Atari games and autonomous driving scenarios to demonstrate the effectiveness of the MDX framework in detecting multiple types of anomalies, thus improving the safety and reliability of RL systems.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. I know the work focuses on deep RL, but some experiments on standard RL environments, like openAI gym, should be reported.\n2. It's better to add some downstream tasks for the offline setting.\n3. Please revise the notation and writings, for example, in the second line of section 4.1, one $s_t$ is bold and another one is not. I assume they are the same $s_t$.", "broader_impact_concerns": "No ethical / broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tpicnE29Pf", "reviewer": "Reviewer_e3pr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper formally studies anomaly detection in RL. I’m not overly convinced by the authors’ claims that there has not been extensive study in these directions. It may just be a matter of semantics, but I believe that there is substantial overlap with considerations within MetaRL and generalization studies that need to be more fully considered as components of anomaly detection in this work. One such paper is Ajay, et al (2022 NeurIPS) that addresses detecting and adjusting for meta RL agents drifting out of distribution in subsequent tasks.\n\nWhile I acknowledge that there are differences between anomaly detection and the above from a fundamental perspective, I do however believe that differentiating or more clearly describing what is meant by the authors regarding generalization vs. anomaly detection. If this paper is solely focused on detection and not mitigating erroneous behaviors when the agent drifts OOD, then that’s fine. I think that we should be more clear here where the paper is directed and what it’s limitations are.\n \nI enjoyed this paper. I am not entirely sure that distance based measures are the appropriate form of determining anomalies but it does appear to be serviceable based on the experimental results. I appreciated that the authors took time to develop multiple approaches for detecting anomalies within the Mahalanobis distance formulation and considered them all equally. I appreciated the extent with which the experiments considered each approach (and appropriate simple baselines) equally. I am however slightly disappointed that there were not additional anomaly detection baselines included in the results from the methods introduced in Section 2. There was a claim that distance based methods are better suited for the envisioned task but it is difficult to gauge whether this claim is satisfied by virtue of the included experiments.\n\nHowever, I found the development of the proposed MDX method to be easy to follow and relatively straightforward. To this end, I think that the paper does a good job introducing the primary contributions even though the communication of them may have been a bit muddled (see the various weaknesses and questions below).\n\nThere are several weaknesses that I noted throughout the paper and I enumerate them below via questions that I seek clarity on.\n\nDoes MDX assume that there are a discrete, finite set of actions? This is an important assumption/limitation that should be mentioned up front. As noted in the footnote on page 5, this can be extended to continuous action spaces but there are possibly computational overheads as well as tradeoffs with the resulting binning strategy to discretize the action space in such settings? Overall, there could be a much clearer description of the types of problems that MDX is envisioned to solve.\n\nOne major weakness of the proposed framework is that there is fairly little maturity about what the intent of an anomaly detector is. How does this change the learning/operating environment of the RL algorithm? There was some slight nod toward this in the introduction but I found that there is some aspect of further discussion and/or development of this idea and where it fits into an intended use case to be lacking.\n\nNot necessarily a weakness but a consideration of potential offline-to-online settings is missing. In this vein, I’m surprised that the authors did not connect their proposed framing within the risk-sensitive RL literature. There is a significant amount of overlap with the terminology and motivations (“unsafe actions”, etc) that exists there. \n\nI think that it needs to be noted more formally that there are distinct approaches under the framework when operating in either offline or online settings and the choice of one form of MDX warrants a different setup, computational burden, etc. This is kind of difficult to wrap my head around. The paper seems to be proposing a single solution (MDX) but then at the end transitions the narrative to be a survey or study of distance based anomaly detection approaches in RL. I think that the paper would be greatly improved if the authors decided on a single narrative. I think that the strongest one would be that the paper is a survey of distance based anomaly detection approaches in RL (both offline and online) rather than attempting to proposed a single method/framework as a \"novel\" contribution.\n\nI appreciated the differentiation made between prior anomaly detection approaches within RL to the proposed distance-based approach. However, how reliable is a distance measure in highly complex, high dimensional state-space domains? This is mitigated somewhat by using dimensionality reduction approaches over the feature space, but I think that this is a consideration about the appropriateness of the proposed anomaly detection framework.\n\nThe paper perhaps could be better structured by introducing a more general overview of how MDX is constructed, building from the foundations of the prior literature and background (Sections 2 and 3), before specifying specific variations on top of the framework for Offline (current Section 4) and Online RL (current Section 5). This is, of course, if the authors decide to maintain their current intended framing of introducing a singular framework (which I do not believe the paper supports at this time). The current structure, with some rewriting, could be maintained if the paper was changed to more appropriate be a study into the effectiveness of different types of distance-based anomaly detection approaches.\n\nI’m not entirely certain that all of Section 4.1 is necessary to motivate and introduce what is currently in Section 4.2. They could each be condensed and combined into a single section. Since there is a clear tonal shift in Section 4.3, I think that the writing could be improved by more directly getting to the conformal, distribution free detection approaches used at the core of what may be the proposed MDX framework (if the narrative structure of the paper is to focus on a singular framework). The optionality of switching between the Chi-Squared thresholding (Gaussian assumption) vs. the conformal approach is not totally clear. I believe that the paper would be improved if one approach is specified and the other is used as a potential ablation to justify the other? Currently, the beginning of Section 4.4 is confusing due to the optionality of nominal MD vs. robust MD, Gaussian assumption vs. conformal detection, etc. \n\nIn online settings, are there steps to re-identify if inliers or outliers may have changed based on the shifting of the policy distribution? I wonder if there is not additional robustness that may be found through the reinvestigation of whether Inlier or Outlier status may have changed with the policy. Of course, this may not be necessary but it could be worth re-visiting? This may not be a relevant question if the moving window casts away data that is \"too old\".\n\nThe moving window estimation section is missing sufficient technical description of what is happening. Is the moving window corresponding with the iteration/time of the policy updates? What happens when the data falls outside that window? Is it cast away? Is there a concern about catastrophic forgetting when the anomaly detection mechanism is only seeded with the most recent data? This is perhaps of a lesser concern when operating in episodic settings with restarts, where the initial state distribution may remain fixed, but this has not been specified clearly by the authors. \n\nIt's unclear what is meant toward the end of Section 5 (regarding Online RL) about pre-training a policy before performing anomaly detection procedures. I think that this is a fairly important detail about how the MDX framework is expected to be evaluated and used that is omitted from this paper. From this, I am fairly uncertain what takeaways I have about the work.\n\nIn Appendix B.4, it is shown that an increase in the number of retained features aids in anomaly detection (not surprising given the high dimensionality of the provided image observations). However, the choice of 50 features is not explained for the main results provided in the paper. Why? Especially since 50 appears to be the lower end of what was tested in the experimental analysis.\n\nIn the CARLA experiments, why are there no learning curve analyses? If this is to be included in the online RL section, and with the prior emphasis on the MDX framework’s adaptations for this learning setting, it feels a bit incomplete to not have some indication of how the learning progresses through time and with the introduction of anomalous observations.\n\nCorresponding references:\n\n#### Ajay, Anurag, et al. \"Distributionally adaptive meta reinforcement learning.\" Advances in Neural Information Processing Systems 35 (2022): 25856-25869.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces an anomaly detection framework based on Mahalanobis Distance based on hypothesis testing for use in Deep RL settings. The proposed anomaly detection framework functions in both offline and online learning settings by virtue of adapting the proposed framework via using moving window estimation and double self-supervised detectors in online settings, where new data is collected through time.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "From the weaknesses and other comments above:\n - Provide a more detailed description of the intended RL problem setting that MDX (and/or all distance-based anomaly detection methods)  is situated.\n - Revise the narrative structure to be either 100% for introducing and testing a new framework or 100% as a study/investigation into distance based anomaly detection.\n - Better framing of anomaly detection vs. RL generalization and/or meta RL considerations of OOD-ness.\n - Discuss how MDX may fit into an offline-to-online learning environment.\n - Provide more information about how the policy is pre-trained in online experiments. What is meant by pretraining? What is the specific set-up for this? \n\nAdditional changes:\nIt would be helpful if summary sentences were included at the end of each paragraph of Section 2, directing the reader to understand how the content relates to the proposed solutions and how the prior work lays the foundation for these advances. The final paragraph “Conformal Prediction and Conformal Anomaly Detection” provides a great example of where this works well.", "broader_impact_concerns": "This paper introduces a preliminary proof of concept about anomaly detection within RL settings. It does not propose a solution to real-world use cases and should not be used as such. I do not have any broader impact concerns in this regard since the methodology and proposed framework has not been rigorously evaluated in sim-to-real settings and I strongly doubt that anyone who was working in these settings would employ the proposed framework without additional safeguards in place.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "j1UTdgEWDU", "reviewer": "Reviewer_8DuC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper does thorough analysis of the MD based anomaly detection methods for both online and offline experiments. And the writing is clear and easy to understand. \n2. The introduction of conformal prediction with distance based outlier detection reduces the assumption of Gaussian distribution, where in many cases Gaussian distribution assumption is actually false. Also as expected, the conformal MD method achieves the best result.\n\nWeaknesses:\n1. In the experiments, it only compares with other distance based methods and among the methods proposed in this paper. But lack experiments about how these distance based methods would compare with other types of state-of-art outlier detection methods.\n2. It is not written clearly how the outlier detection is related to the policy performance improvement. I see in the online RL experiments part, some results about rewards are shown. But not sure how the paper wants to show the correlations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is about a crucial issue of anomaly detection in deep reinforcement learning. It proposes methods to detect outliers in deep RL based on Mahalanobis Distance(MD), and improve it to robust MD-based detection and conformal MD-based detection. While many distance based outlier detection metrics rely on a Gaussian assumption to construct the detection threshold, this paper utilizes conformal prediction to get rid of the Gaussian assumption, which brings the theory closer to reality without many constraints. This paper considers the problem of anomaly detection in both offline and online RL settings. The paper also includes experiments in each setting to verify the effectiveness of the MD based outlier detection methods, within which the conformal MD method achieves the best detection performance in most scenarios.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Requested changes:\n1. It will be more convincing to add other state of art outlier detection methods to compare. Outlier detection is a common problem involved in many transfer learning algorithms, like this paper “Bridging Theory and Algorithm for Domain Adaptation”. So for example, how the distance based metric is compared with neural network based discriminator to detect outliers?\n2. Now most of the outliers/anomalies are artificially made, like adding Gaussian noises and so on. How would this outlier detection method perform with some real domain shift datasets? \n3. It would be helpful to declare how the outlier detection is related to policy performance improvement.", "broader_impact_concerns": "Not related.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes anomaly detection methods for RL algorithms, which are able to cast an alert when an RL algorithm encounters an unfamiliar state. The core technical framework is based on Mahalanobis distance, and a number of different variants are investigated, spanning the settings of offline, online, Gaussian, and distribution-free. It is also shown that during RL training, using the proposed anomaly detector to filter the encountered states can increase the training robustness. Extensive experiments are conducted to demonstrate the effectiveness of the proposed methods. 

The reviewers all agree that the paper presents valuable contributions to the field. The technical developments are solid, and the writing is clear. There were initially some concerns on the discretization of the action space and the associated computational cost, but these were cleared during the rebuttal. As suggested by the reviewers, more experiments were added in the revision. 

A remaining concern after the rebuttal is that more baselines are suggested for the experiments, and this is essentially related to the clarity of the paper's scope. The paper studies two objectives, anomaly detection, and using the proposed detector to improve the training robustness. While the first objective is clear and self-contained, the second objective possesses some intuitive similarity to certain related topics, such as "risk-sensitive RL" or "safe RL", therefore the reviewers would appreciate more baselines of such types. However, a consensus is that this is good to have but not a must. The current form of the paper is of high quality, and meets the acceptance criterion. 

Therefore the recommendation is to accept the paper as is. The authors are encouraged to further sharpening the paper's scope and revising the related exposition, particularly in the section of experiments, during the camera-ready revision.
Decision: Accept (poster)