Task-Driven Out-of-Distribution Detection with Statistical Guarantees for Robot Learning
Authors: Alec Farid
Sushant Veer
Anirudha Majumdar
Venue: CoRL2021 Poster (2021)
Keywords: Out-of-distribution detection
generalization
PAC-Bayes

Abstract: Our goal is to perform out-of-distribution (OOD) detection, i.e., to detect when a robot is operating in environments that are drawn from a different distribution than the environments used to train the robot. We leverage Probably Approximately Correct (PAC)-Bayes theory in order to train a policy with a guaranteed bound on performance on the training distribution. Our key idea for OOD detection then relies on the following intuition: violation of the performance bound on test environments provides evidence that the robot is operating OOD. We formalize this via statistical techniques based on p-values and concentration inequalities. The resulting approach (i) provides guaranteed confidence bounds on OOD detection, and (ii) is task-driven and sensitive only to changes that impact the robot’s performance. We demonstrate our approach on a simulated example of grasping objects with unfamiliar poses or shapes. We also present both simulation and hardware experiments for a drone performing vision-based obstacle avoidance in unfamiliar environments (including wind disturbances and different obstacle densities). Our examples demonstrate that we can perform task-driven OOD detection within just a handful of trials. Comparisons with baselines also demonstrate the advantages of our approach in terms of providing statistical guarantees and being insensitive to task-irrelevant distribution shifts.

Task-Driven Out-of-Distribution Detection with Statistical Guarantees for Robot Learning
Alec Farid" Sushant Veer* Anirudha Majumdar
Department of Mechanical and Aerospace Engineering, Princeton University {afarid, sveer, ani mmajundarj ] jpprinceton edu
Abstract: Our goal is to perform out-of-distrribution (OOD) detection, i.c., to de- tect when a robot is operating environments that are drawn from a different distribution than the environments used to train the robot. We leverage Proba- bly Approximately Correct (PAC)-Bayes theory in order to train a policy with a guaranteed bound on performance on the training distribution. Our key idea for OOD detection then relies on the following intuition: violation of the performance bound on test environments provides evidence that the robot is operating OOD. We formalize this via statistical techniques based on p-values and concentration inequalities. The resulting approach (i) provides guaranteed confidence bounds on OOD detection, and (ii) is task-driven and sensitive only to changes that impact the robot's performance. We demonstrate our approach of grasping objects with unfamiliar poses or shapes. We also present both sim- ulation and hardware experiments for drone performing vision-based obstacle avoidance in unfamiliar environments (including wind disturbances and different obstacle densities). Our examples demonstrate that can perform task-driven OOD detection within just a handful of trials. Comparisons with baselines also demonstrate the advantages of our approach in terms of providing statistical guar- antees and being insensitive to task-irrelevant distribution shifts.
simulated example
Keywords: duttt--disttibution detection, generalization, PAC-Bayes
Introduction
Imagine drone trained to perform vision-based navigation using a dataset of indoor environments and deployed in environments with varying wind conditions, obstacle densities, or lighting (Fig. 1). Similarly, consider a robot arm manipulating new set of objects in a new city. State-of-ther oh-therrrt art techniques for lcarning- based control of robots typically struggle to generalize to such out-of distribbbtt (OOD) environments. This lack of OOD generalization is par- ticularly pressing in -afety-critical settings, where the price of failure is high. In this work, we focus on the problem of autonomously detecting when robot is operating in environments drawn from different distribution than the one used to train the robot. This ability to perform OOD detection has the potential improve the safety of robotic systems operating in OOD environments. For example, a drone operating in a new set of environments could either deploy a highly conservative policy or cease its operations altogether. In addition, OOD detection can also allow the robot improve its policy by re- re-training using additional data collected from the new environments.
autonomous vehicle deployed
There are two important desiderata that OOD detection approaches for anfety-critical robotic sys- tems should idcally satisfy. First, we would like to develop OOD detection techniques with guar- anteed confidence bounds. Second, we would like our OOD detectors to be task-driven and only sensitive to task-relevant changes in the robot's environment. As an example, consider again the drone navigation setting in Fig. 1 and suppose that the robot's policy is insensitive to changes in color and lighting. Here, the robot's OOD detector should not trigger even if the robot is operating in environments with different color/lighting and should only trigger if there are task-relevant vari- ations (e.g.. variations in the obstacle density). Unfortunately, current approaches (Sec. 2) do not typically satisfy both desiderata; they are often based on heuristics and not task-driven in general.
Statement of Contributions. We develop task-driven OOD detection techniques with statistical guar- antees correctness. To this end, we make three specific contributions (see Fig. I for overview).
Equaal Contribution
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Training Distribution Environments
Test Distribution Environments (Wind More Clutter)
Train pollcy
Bound violation Implies OOD with guaranteed confidence
performance bound
Figure 1: A schematic of our overall approach. We learn policies with guaranteed bounds on expected perfor- mance the training distribution. Violation of this bound during deployment implies that the robot is operating OOD (with high confidence). present simulation experiments manipulation and navigation settings, and hardware experiments for drone navigating in new environments with varying wind conditions and clutter.
Given dataset of environments drawn from an (unknown) training distribution, we develop pipeline based on generalization theory for training control policies with guaranteed bound on performance (a bound on the expected cost of the policy on the unknown training distribu- tion). Specifically, we leverage recently-developed derandomized Probably Approximately Cor- rect (PAC)-Bayes bounds that are well-suited to enable OOD detection (Sec. 4.1).
We develop two OOD detection techniques with complementary statistical interpretations based on the following intuition (Sec. 4.2): if the costs incurred when the robot is deployed in new environments violate the bound on the policy's performance, then this indicates that the robot operating OOD. We formalize this via p-values and concentration incqualities and provide statis- tical guarantees on the false positive rate OOD detection. Since our OOD detection scheme leverages the costs incurred in new environments, it is only triggered by task-relevant changes.
We demonstrate our approach on two simulated examples (Sec. 5): (i) a robotic manipulator grasp- ing new set of objects, and (ii) drone navigating new set of environments. Comparisons with baselines demonstrate the advantages our approach in terms of providing statistical guarantees and being insensitive lask-irrelevant shifts. We also present thorough set of hardware exper- iments for vision-based drone navigation with varying wind conditions and clutter (Fig. 1). Our experiments demonstrate the ability of our approach to perform task-driven OOD detection within just handful of trials for systems with complex dynamics and rich sensing modalities.
2 Related work
Anomaly/OOD detection in supervised learning. Anomaly detection in www-dimeniional signals has been well-studied in the signal processing literature (see [1] for a review). Recent work in machine learning has focused on OOD detection for high-dimensional inputs (e.g.. images) in su- pervised learning settings (see [2] for review). Popular approaches use threshold-based detectors the output distribution of given pre-trained neural classifier [3, 4, 5]. Other methods use a specific pipeline for training a neural network in order to improve OOD detection on test samples [6. 7, 8]. However, these methods are often susceptible to adversarial attacks [9]. Thus, approaches also
for addressing adversarial data have been developed [10, 11]. Some of these approaches able to provide theoretical guarantees performance on adversarial data [9, 12, 13). Other methods provide PAC-style statistical guarantees [14, 15] p-values [16]. However, these methods typically focus on supervised learning settings and often require specific network outputs (e.g.. softmax) that incompatible with con-classification tasks. In contrast, focus OOD detection for policy learning settings robotics and do not make assumptions about the specific structure the policy.
Task-driven OOD detection. The methods above are aimed at detecting any distributional shift in the data and can be sensitive even to task-irrelevant shifts (i.e., ones that not impact performance) we demonstrate in our experiments (Sec. 5), A recent approach determines an estimate of input

---

atypicality for pre- vee-triied networks and uses it as an OOD detector in supervised learning settings [17]. Recent methods have also been developed specifically for reinforcement learning (RL) [18, 19, 20, 21, 22]. In particular, [22] presents a general task-driven approach for OOD detection on sequential rewards, which is optimal in certain settings. However, neither this method nor others in the RI. context provide statistical guarantees on detection. We propose an OOD detection framework which is both task-driven and provides statistical guarantees by leveraging generalization theory.
Generalization theory. Generalization theory provides a way to learn hypotheses (in supervised leaming) with a bound on the true expected loss on the underlying geta-generating distribution given only finite number of training examples. Original frameworks include Vapnik- Chenik--heroonniis (VC) theory [23] and Rademacher complexity [24]. However, these methods often provide vacu- ous generalization bounds for high- high-dimensional hypothesis spaces (e.g., neural networks). Bounds based on PAC-Bayes generalization theory [25, 26, 27] have recently been shown to provide strong guarantees in a variety of settings [28, 29, 30, 31, 32, 33], and have been significantly extended and improved [34, 35, 36, 37, 38, 39]. PAC-Bayes has also recently been extended to learn policies for generalization to novel environments [40, 41, 42, 43]. In the present work, we leverage recently-proposer derandomized PAC-Bayes bounds [44]; this framework allows us to train a single deterministic policy with guaranteed bound on expected performance on the training distribution (in contrast to [40, 41, 42, 43], which train stochastic neural network policies). This forms the basis for our OOD detection framework: by observing violations of the PAC-Bayes bound on test environments, we are able to perform task-driven OOD detection with statistical guarantees.
robots with guarantees
3 Problem formulation
Dynamics and environments. Let 8t+1 = fE(SE, 0+) describe the robot's dynamics, where 8, E S T is the state of the robot time-step t. ar E A R is the action, and E the environment that the robot is operating in. "Environment" here broadly refers to factors that are ex- ternal to the robot, e.g., a cluttered room that a drone is navigating, disturbances such as wind gusts, or object that manipulator is grasping. The dynamics of the robot may be nonlinear/hybrid. We denote the robot's sensor observations (e.g., RGB-D images) by Or E O C B
Cost functions. incurred by (deterministic) policy æ when deployed in environment E' over finite time horizon T. The policy π E is a mapping from (histories of) sensor observations to actions (e.g., parameterized using a neural network). In the context of obstacle avoidance, the cost could capture how close the drone gets to an obstacle; in the context of grasping, the cost could be 0 if the robot successfully lifts the object or 1 otherwise. We assume that the cost bounded; without further loss of generality, we assume (F(T)) [0, 1]. We also assume that the robot has access to the cost Cε(T) after performing a rollout on E (i.e., at the end of an episode of length T). This is a relatively benign assumption in robotics contexts since the cost function often has physical meaning and can be measured by the robot's sensors. For example, a drone equipped with depth sensor can measure the smallest reported depth value during its operation in an environment, and manipulator equipped with camera and/or force-torque sensor can measure if it successfully grasped an object. We make no further assumptions on the cost function (c.g.. we do not assume continuity, Lipschitzness, etc.).
The robot's task is encoded via a cost function and we let CE(T) denote the cost
Training and testing distribution. We assume that the robot has access to a training dataset S = {Ei...., Em } of m environments drawn i.i.d. from a training distribution D, i.e. S ~ D After training, the robot is deployed on environments S' = {EE E,) drawn from a test distribution D': S' ~ D'n Importantly, we do not assume any explicit knowle hloweedge&o of D, D' or the space & of environments. We only have indirect access to D, D' in the form of the finite training datasets S'
Goal: task-driven OOD detection. After being deployed in (a typically small number of) en- vironments in S', the robot's goal is to detect if these environments were drawn from a different distribution than the training distribution (i.e., if D' is different from perform task-driven OOD detection. In particular, our OOD detector should only trigger if:
D). Moreover, our goal is to
CD = E'D
(T) > Cp(T) = E CB(T))
(1)
E~D
Thus, our OOD detector should be insensitive changes in the environment distribution that do not adversely' impact the robot's performance. This is a challenging task since we only assume
'The OOD detection scheme we present can also be modified to detect if the expected cost on D' is smaller than the expected cost on D; however, we focus on the other case since that of greater practical interest.
3

---

access to finite number of environments from D and D'. Moreover, our goal is to develop an OOD detection framework that broadly applicable in challenging settings involving nonlinear/hybrid dynamics, rich sensing modalities (e.g., RGB-D), and neural network-base policies.
Approach
Our overall approach is illustrated in Fig. 1. First, we train a policy with an associated guarantee on the expected cost on the training distribution D (Sec. 4.1). We then apply our OOD detection scheme which formalizes the following intuition: violation of the bound during deployment implies (with high confidence) that the test distribution D' is OOD in task-relevant manner (Sec. 4.2).
4.1 Policy training via derandomized PAC-Bayes bounds
Given a training dataset S = {Ei,..., Em } of m environments drawn i.id. from the train- ing distribution D. our goal to learn policy π with a guaranteed bound on the expected cost CD(T) = (E Since our OOD detection scheme will rely on violations of the bound, it important to obtain bounds that are as tight as possible. In this work, we utilize the Probably Ap- proximately Correct (PAC)-Bayes framework [25, 26, 27] to train policies with strong guarantees. derandomized PAC-Bayes bounds [44], which
More specifically, we leverage recently-develope are well-suited to the OOD detection setting (as we explain further below).
PAC-Bayes applies to settings where one chooses distribution over policies (e.g.. a distribution over weights of a neural network), and learning algorithms that have the following structure: (I) choose a "prior" distribution Po over the policy space II before observing any data (this can be used encode domain/cxpert knowledge); (2) obtain a training dataset S and choose posterior distribution P over the policy space IT. Let P be the output of an algorithm A which takes Po and $ as input. Denote the cost incurred by a policy on the training environments in S as Cs(x) == i EEES CF(T). The in following result is our primary theoretical tool for training policies with bounds on performance.
Theorem For any distribution D, prior distribution Po, E (0, 1), cost bounded in [0, 1], m 2 8, and deterministic algorithm A which outputs the posterior distribution P, we have the following:
(C((T) s SD( 21-8
P (Sx)) xP)
(2)
where ST( Cs(x) + VR. R = (D2(PIP6) + In T (5/2)3 \ /2mm) and D2 is the Rényi
Divergence for a = 2 defined as: (P1(P)) = In (E Po(T)
2
Proof. The proof is in App. A.I. We use [44, Theorem 2]. general pointwise PAC-Bayes bound. We perform the reduction from supervised learning to policy learning presented in [40].
O
This result allows us to obtain policies with guaranteed bounds on the expected cost. In particular, we can search for a posterior P in order to minimize the sum of the training cost and the "regularizer" VR. We describe such training methods via backpropagation and blackbox optimization below. Sampling from the resulting posterior P' provides policy with bound on CD(r) that holds with high probability (over the sampling of the training dataset S and the policy T).
Recent work has demonstrated the effectiveness of PAC-Bayes to provide strong bounds for deep neural networks [28, 31, 33] and specifically for policy learning [40, 41, 42, 43). However, the bounds used by these approaches do not provide a viable approach for performing OOD detection. The approaches are based on traditional PAC-Bayes bounds, where distribution P over policies (e.g., distribution over neural network weights) is chosen; the resulting bound is on B.-PCD(T) instead of CD(T). Thus, given a test dataset S" of environments, many policies from the distribution P must be sampled in order to evaluate/bound the expected cost on S'. This is not feasible in an OOD detection setting, where there is single execution on the test environments. Our use of the derandomized PAC-Bayes bound in Theorem avoids this issue since we can bound (D() for particular policy sampled from P.
We provide approaches for optimizing the bound provided in Theorem 1 using backpropagation (App. A.4) and Evolutionary Strategies (ES) [45] (App. A.5). Since Theorem requires a deter- ministic training algorithm, we fix the random seed for stochastic training methods. This makes the algorithm deterministic as the same input will always produce the same output. We choose multi- variate Guassian distributions with diagonal covariance diag(s), i.c., P. NN, diag(s)). for the

---

posterior P and prior Po distributions. Further, let &b := (1, log s); we use the shorthand Ny for Ni, dia We denote un with weights ~ Np as a parameterization of the robot's 's policy (e.g,, neural networks with weights w). After training, we sample and fix a w from the trained posterior for deployment on test environments, We then compute the PAC-Bayes upper bound (T,
4.2 Task-driven OOD detection with statistical guarantees
We now tackle the problem of OOD detection as defined Sec. 3. The PAC-Bayes training pipeline from Sec. 4.1 produces policy x and an associated bound (T S) on the expected cost Cp (T) that holds with probability 1 5 over the sampling of the training dataset S ~ DD and the policy л ~ P. Our key idea for OOD detection is that if our PAC bound (T S) is violated by π in the test environments S' (drawn from the test distribution D'), then this indicates that the test environments are OOD. We formalize this intuition below using two popular frequentist statistical inference tools hypothesii testing via p-value and confidence interval overlap.
Method 1: Hypothesis testing
In accordance with our notion of task-driven OOD detection (1) described in Sec. 3, let Hy be the null-hypothesis which claims that (TT) S CD(z) and let H₃ be the alternate hypothesis which claims that CD (T) CD(m for given policy Then, to perform hypothesis test, compute the p-value and check if it drops below a significance level a E (0,1). which is chosen before looking the data. The p-value bounds the probability of drawing test environments with average cost larger than the observed test cost Cg (T). assuming that the null-hypothesis holds. In the event that the p-value is smaller than a, we can conclude that under the null-hypothesis the observed test dataset S' had a very small probability of being drawn; therefore, the null-hypothesis Ho can be rejected. A mathematically precise definition of the p-value is given as follows.
Definition 1 (adapted from [46]) Let D'' be the test distribution and S' D'n be an observed dataset. Let π be the robot's 's control policy. Then, the p-value is defined as:
p(S") P SNDIM
(Cg(T) 2 (T) Ho)
(3)
Note that S is any dataset of cardinality n drawn i.i.d. from the test distribution D'n, while S is the observed test dataset which we must use for inference. We present an upper bound on the p-value by leveraging the PAC- PAC-Bayes generalization bound (Theorem 1). This upper bound holds with probability 1 - 5 (over the sampling of S and x).
Theorem
Let be the training distribution and P be the posterior distribution on the space of policies obtained through the training procedure described in Sec. 4.1. Let S' ~ DDm be a test dataset, p(S') be the p-value for S' defined in Definition! I, and E (0, Then,
P
[p(5") I exp( exp(-2n7(S))) 2 2211 1-8,
(4)
(Dm))<(DXXP))
where T(S)= == max(Cs (T) - ST,
Proof. The proof relies on Hoeffding's inequality and is provided in App. A.2.
Theorem 2 provides an upper bound on the p-value which holds with high confidence. If the upper bound is below the significance level then with high confidence we can say that the p-value is below o;; thereby, Theorem 2 facilitates OOD detection through hypothesis testing.
Method 2: Confidence interval on the difference in expected train and test costs
We now present another method for performing task-driven OOD detection (Sec. 3 and Eq. (1)) by lower bounding CD (m) - (T) with high probability. If the lower bound is positive, then with high confidence we can conclude that the robot operating (task-relevant) OOD environments as the test cost is larger than the train cost. Furthermore, this lower bound serves as measure of how "far" the two distributions are from the perspective of the task.
Theorem 3 Let D be the training distribution, 'D' be the test distribution, and P be the posterior distribution on the space of policies obtained through the training procedure described in Sec. 4. I.
Let 8, 8' E 1) such that 8 8' < 1, 1.y:= == /
In (1/8"). and AC == Cs-(TT) - 7 Cs(a, S). Then,
2n
(CD - (p)m) Z AC| 2 1 6-8
(5)
(Dm
5

---

Proof. A detailed proof of this theorem is provided in App. A.3.
To use Theorem 3 for detecting if test dataset S" is OOD, we first choose 8 and 8' obtain desired confidence level - 5 8' for our OOD detector. Using these constants and the policy π drawn from P trained using the PAC-Bayes upper bound, we then compute AC. If AC > O, then with confidence at least 1 8 8' we can claim that CD > CD. and therefore, with the same level of confidence the test dataset S' must be OOD according to our task-relevant notion in (I). Furthermore, we obtain a guaranteed upper bound of 5 + 8' on the false positive rate of our OOD detector, i.e., the rate at which our detector will misclassify training distribution data as OOD.
5 Examples
We demonstrate the ability of our approach to perform task-driven OOD detection with guar- anteed confidence bounds on two simulated examples: a manipulator grasping a new set of objects and drone navigating a new set of environments. For the navigation task, we compare our methods with popular OOD detection baselines. Our code is available at: https://github. com/irom-lab/eestee..a..oaeaa Rew.aastte..o Detectiion
5.1 Robotic grasping
Overview. We use the Franka Panda arm (Fig. 2(a)) for grasping objects in the PyBullet simulator [47] and build upon the open-source code provided in [42]. The robot employs a vision-based control policy that uses a depth map of the object obtained from an overhead camera and returns open- loop action a == (ar, y, z: 0) which corresponds to the desired grasp position and yaw orientation of the gripper. We train the manipulator to grasp mugs placed in (2) poses drawn from a specific distribution. Then, we demonstrate the efficacy of our OOD detection framework by (i) gradually modifying the distribution the mug poses and (ii) changing the objects from mugs to bowls.
Control policy. The control policy is a deep neural network (DNN) which inputs a 128 x 128 depth map of the object and a latent state E IR sampled from multivariate Gaussian distribution No: with a diagonal covariance, and outputs an open-loop grasp action a: see Fig. 4 in App. A. for the policy. In [42], the distribution N₁. on the latent space encodes prior domain/expert knowledge.
Training. Mugs from the ShapeNet dataset [48] are randomly scaled in all dimensions generate a training dataset S of 500 mugs. If the robot is able to lift the mug by 10 cm and the gripper palm does not contact it, then we consider the rollout successful and assign a cost of O; otherwise the cost is set to 1. In training, we optimize the distribution N₂. on the latent space to minimize the PAC- Bayes upper bound provided in Theorem I using Alg. 2, while the weights of the CNN and MLP networks in Fig. 4 in App. 1. remain fixed. The prior Ne, is chosen as the normal distribution with zero mean and identity covariance. A policy π is sampled from the trained posterior Ny and the PAC- MAC-Bayes bound for this policy computed as (z S) 0.1 with 5 = 0.01.
OOD detection results. We perform OOD detection using the two methods presented in Theorem 2 and Theorem 3. For detection with p-value, we choose a significance level detection using AC we choose a confidence level of 95%, i.e., 8+0' = 0.05, which ensures that the false- possitvose rate of our detector is no greater than 5%. We perform two experiments to demonstrate the efficacy of our approach for OOD detection. First, we make the distribution on the mug's ' initial placement progressively more challenging; see App. A. for the exact distributions. For each distribution, we sample 20 test datasets of cardinality 10 and compute our OOD indicators: (i) the lower bound on - p (where p the p-value) and (ii) AC using Theorem 3. Fig. 2(b) plots the mean (dashed line) and one standard deviation spread (shaded region) for the OOD indicators as a function of Cp - Cp (estimated via exhaustive sampling). Note that we plot AC + 0.95 so that the OOD threshold is the same (0.95) for both methods. As the cost of the policy deteriorates distributions our OOD indicators reliably increase, capturing the shift of the test distributions away from the training distribution. In the second experiment, we change the objects that the manipulator must grasp from mugs to bowls. Fig. 2(c) shows that with a small test dataset S' of cardinality 5, both our approaches detect OOD when bowls are used (red curves). As expected, our OOD detectors are not triggered for mugs (blue eurves), which are drawn from the training distribution.
95%, while, for
test
5.2 Vision-based obstacle avoidance with a drone
Overview. In both the simulation and hardware portions of this example, we aim to avoid an obstacle field with the Parrot Swing drone; this is an agile quadrotor/fixed-win wwin hybrid drone shown in Fig. 1.

---

1.6-
OOD indicator
OOD indicator
00D threshold
-05
Mug: 1-p
000 threshoid
-16
Mug: &c 0.95
1
AC 0.95
Mugs
Dowts
Estimated Cyrm) Culm)
Cardinality of Test Data
(c)
Figure 2: OOD detection for grasping. (a) Franka Panda amm in PyBullet grasping a mug (left) and a bowl (right). (b) Performance of our OOD detectors for different distributions on mug placement. Both our ap- proaches perform similarly and the OOD indicators increase monotomically with (Dr(T) - CD(T). (c) Com- parison of our OOD detectors for grasping mugs and grasping bowls. Both our approaches detect OOD using a small number of test environments (just 4) for bowls and do not detect OOD for mugs (as expected).
We train a DNN control policy in a simulation setup based on the hardware system shown in Fig. 1. The policy takes in 50 x 50 depth image and outputs softmax corresponding to a of pre- computed motion primitives with the goal of avoiding obstacles by the largest distance. Since we designed the simulation portion of this example with application to hardware in mind, created motion primitives by capturing (with Vicon motion tracking system) the trajectories of open-loop control inputs, which result in different maneuvers; the two images in Fig. I represent two of these trajectories. The use of motion primitives allows us to perform accurate sim-to-real transfer (since the motion primitives are recorded directly from the hardware system).
have
Training, Environments consist of a set of randomly placed cylindrical obstacles. We record the minimum distance dmin from the obstacles (as recorded by the robot's 120 field of view depth sensor) and assign a cost of max(0, prior to assign larger values to motion primitives which achieve larger distance from obstacles. See App. A.6.2 for further details the training procedure for the prior. We then use another 10,000 environments to train the posterior distribution using Alg. 1. We sample policy π from the trained posterior and compute the PAC-Bayes bound (TT, S) = 0.222 for ó = 0.01.
duin ). Using 10,000 training environments S, we train
Simulation Results
We compare our task-driven OOD detection approach with two baselines: (i) maximum softmax probability (MSP) [3] (an effective and popular baseline for OOD detection). and (ii) MaxLogit [5] (a recent state-of-the-an OOD detection baseline). We note that these baselines are specifically designed for networks which output categorical distributions, and thus provide strong benchmarks. We determine the threshold for detection for the baselines by choosing a false positive rate of 5% for the training environments. However, note that the baselines do not provide any guarantees; they may violate the false positive rate even on new environments drawn from the training distribution (as observe in experiments). We select p- p-vuuue of 0.05 the indication that an environment is OOD, and a guaranteed talse-positive rate of + 8' = 5% for the confidence interval method. Each method receives dataset S' of 10 environments. We use the average outputs of the baselines on the 10 environments for OOD detection. We generate OOD environments of varying difficulty by changing the number of obstacles and the maximum or minimum gap-size between obstacles.
Varied enviromment difficulty. We sample 2000 datasets S' (of size 10) for each setting and plot the proportion of these detected as OOD for all methods in Fig. 3(a). At 0 on the x-axis of this plot, we draw datasets from the training distribution. All other environments are OOD. We note that the p-value and confidence interval-based methods perform similarly. The plot demonstrates our ability to perform uask-driven OOD detection. In particular, we only detect the "harder" distributions (i.c., environments with higher increased cost) as OOD, while the baselines are triggered easier distributions. In addition, neither baseline provides any guarantees and both violate the 5% false-positive rate on the training distribution. Our methods have low false positive rate, albeit at the cost of conservatism for moderate distribution shifts.
for the
Task-irrelevant shift. We also compare the baselines with our methods on a distribution where environments consist of (uniformly) randomly located obstacles. In this setting, the control policy achieves a near-identical expected cost CD (T) (as estimated by exhaustive sampling of environ- ments) to the expected training cost (D(T)) (in particular, CD((m) CD(T)) -0.02). For this
setting, MSP [3] classified 7% of test datasets as OOD and MaxLogit [5] classified 50.9% as
7

---

Proportion detected as 00D
1.0
1.0
0.8
OOD indicator
0.8
0.4
0.4
1-P
AC
0.2-
000 threshold
0.0
MaxLogit
AC + 9.
-0.2
0.0
0.2
0.4
0.6
0
25
50
75
100
Estimated Cp(n) Cp(m)
Percentage max wind disturbance
(a)
(b)
Figure 3: OOD Detection on settings variable difficulty. (a) Comparison of the performance of our OOD
detectors with baselines MSP [3] and MaxLogit [5]. Both our methods do not detect OOD when CD.(T) < (T( whereas the baselines detect these task irrelvvat shifts in the environment. For Cpr(T) >> CD(x) the baselines detect higher proportion of datasets OOD for smaller values of Cpr(m) -n (Tt) compared to methods. (b) Comparison of our OOD detectors on the Parrot Swing hardware for increasing wind disturbance. Both OOD indicators increase monotonically with C (T) and are able detect OOD at 100% wind.
OOD. Thus, the baselines triggered by task-inrelevant shift in the distribution. In contrast, methods had an OOD detection rate of 0.1% % in this setting.
Hardware Results
We use a Parrot Swing drone for the hardware experiments (Fig. 1). We simulate a depth sensor for the drone (as if the sensor was mounted on the drone) by generating a synthetic depth image using the positions of objects from the Vicon motion capture system. We do not provide any other information to the policy (e.g., obstacle positions). We generate each environment the same way as in simulation, and then place the weal-world obstacles in the generated locations.
Varied environment difficulty and wind disturbances. We deploy the policy trained in simu- lation on three kinds of OOD environments in hardware: (i) environments with a smaller num- ber of obstacles ("easier" environments), (ii) environments with smaller gaps between obstacles (i.e., "harder" environments), and (iii) environments with wind generated using a fan (Fig. right) with the same obstacle distribution as training. Videos from each setting are available at https:/ hrrps://yotuu. be/VxKCA aih8M. For each setting, we run trials on the hardware and use this for OOD detection. As expected, our OOD detectors are not triggered by the easier environments. For the harder environments, we compute
aaMMM.
p 2 0.81 and AC = -0.11. Results from the windy environments are shown in Fig. 3(b) for increasing values of wind (up to about 5 m/s). We note that the sim- ro-l distribution shift (corresponding to the zero wind case) is not viewed as being OOD in task-relevan manner by approaches. Both our approaches assign increasingly high confi- dence as the wind speed is increased. We note that disturbances such as wind cannot be detected via the depth image given to the robot's policy. Thus any OOD detection technique which relies solely on the output of the policy, such as MSP [3] and MaxLogit [5], would be unable to detect these environments as OOD.
Conclusion
We have presented framework for performing task-driven OOD detection with statistical guaran- tees. Our approach uses PAC-Bayes theory to train a policy with a bound on the expected cost on training distribution. We then perform OOD detection on test environments by checking for vi- olations of the bound (using p-values and concentration inequalities). Our simulated and hardware experiments demonstrate the ability of our approach to perform OOD detection within a handful of trials. Comparisons with baselines also demonstrate two advantages: OOD detectors (i) are only sensitive to task-relevant distribution shifts, and (ii) provide statistical guarantees on detection.
Challenges and future work. The approach we present here allows us to bound the false positive rate, An interesting theoretical question whether we can also bound the false negative rate. would also be of practical interest to extend our approach to settings where the robot encounters environments in an online manner (instead the batch setting we consider here). Another partic- ularly exciting direction is to develop versions of our approach that are more proactive; instead of having to incur costs on the test environments, one could potentially perform OOD detection based on predicted costs (thus avoiding the need to potentially fail on the test environments).
8

---

Acknowledgments
The authors were supported by the Office of Naval Research [N00014-21-1-2803 N00014-18-1- 2873], the NSF CAREER award [2044149], and the Toyota Research Institute (TRI). This article solely reflects the opinions and conclusions of its authors and not ONR, NSF, TRI any other Toyota entity.
References
[1] M. Basseville. Detecting Changes in Signals and Systems A Survey. Automatica, 24:309- 326, 1988.
[2] L. Ruff, J. R. Kauffimann, R. A. Vandermeulen, G. Montavon, W. Samek, M. Kloft, T. G. Dietterich, and K.-R. Müller. A Unifying Review of Deep and Shallow Anomaly Detection. Proceedings of the IEEE, 109(5):756 (95,6-999, 2021.
[3] D. Hendrycks and K. Gimpel. A Baseline for Detecting Misclassified and Examples in Neural Networks. Proceedings of the International Conference on Learning Rep- resentations, 2017.
[4] S. Liang. Y. Li, and R. Srikant. Enhancing the Reliability of Out-of-Distribuuie Image Detec- tion Neural Networks. Proceedings of the International Conference on Learning Represen- lations, 2018.
[5] D. Hendrycks, S. Basart, M. Mazeika, M. Mostajabi, J. Steinhardt, and D. Song. Scaling Out- of-Distribution Detection for Real-World Settings. arXiv preprint arXiv: 191..1991111111 2020.
[6] T. DeVries and G. W. Taylor. Learning Confidence for Out-of-Distribution Detection Neural Networks, arXiv preprint arXiv::8022 0e0.11002.00.... 2018.
[7] K. Lee, H. Lee, K. Lee, and J. Shin. Training confidence-calibrated classifiers for detecting out-of-distributio samples. Proceedings of the International Conference on Learning Repre- sentations, 2018.
[8] D. Hendrycks, M. Mazeika, and T. G. Dietterich. Deep Anomaly Detection with Outlier Ex- posure. Proceedings of the International Conference on Learning Representations, 2019.
[9] J. Chen, Y. Li, X. Wu, Y. Liang, and S. Jha. Robust -utto--dist Detection for Neural Networks. arXiv preprint wrXiv:2003.09711 2020.
[10] K. Lee, K. Lee, H. Lee, and J. Shin. A Simple Unified Framework for Detecting Out-of- Distribution Samples and Adversarial Attacks. Proceedings of the Conference on Advances in Neural Information Processing Systems, 2018
[11] Bitterwolf, A. Meinke, and M. Hein. Certifiably Adversarially Robust Detection of Out-of- Distribution Data. Proceedings of the Conference on Advances in Neural Information Process- ing Systems, 2020.
[12] A. Meinke and M. Hein. Towards Neural Networks That Provably Know When They Don't Know, Proceedings of the International Conference on Learning Representations, 2020.
[13] A. Meinke, J. Bitterwolf, and M. Hein. Provably Robust Detection of (almost) for free. arXiv preprint (tl::21000002660. 2021.
Data
[14] M. A. Siddiqui, A. Fern, T. Dietterich, and Das. Finite Sample Complexity of Rare Pattern Anomaly Detection. Proceedings of the Conference on Uncertainty in Artificial Intelligence, 2016.
[15] S. Liu, Garrepalli, T. G. Dietterich, Fern, and Hendryeks. Open Category Detection with PAC Guarantees. Proceedings of the International Conference on Machine Learning, 2018.
[16] S. Bates, E. Candès, Lei, Romano, and M. Sesia. Testing for Outliers with Conformal p-values, arXiv preprint arXiv:2104.08273 2021.

---

[17] A. Sharma, N. Azizan, and M. Pavone. Sketching Curvature for Efficient cut-of-Distribution Detection for Deep Neural Networks. arXiv preprint arXiv:2102.12567, 2021.
[18] A. Sedlmeier, T. Gabor, T. Phan, L. Belzner, and C. Linnhoff-Popien. Uncertainty- Based Out-of-Distribution Classification in Deep Reinforcement Learning. arXiv preprint arXiv:2001.00496, 2019.
[19] A. Sedlmeier, R. Müller, S. Illium, and C. Saanooff-Popien Policy Entropy for Out-of- Distribution Classification. Artificial Neural Networks and Machine Coneming-ICANM, 2020.
[20] F. Cai and X. Koutsoukos. Real-time Out-of-distribution Detection in Learning-Fnabled Cyber-Physiee Systems. Proceedings of the International Conference on tems, 2020.
Sys-
[21] T. Wu and J. Ortiz. RLAD: Time Series Anomaly Detection through Reinforcement Learning and Active Learning. arXiv preprint arXiv:2104.00550 2021.
[22] I. Greenberg and S. Mannor. Detecting Rewards Deterioration in Episodic Reinforcement Learning. Proceedings of the International Conference Machine Learning, 2021.
[23] V. N. Vapnik and A. Y. Chervonenkis. On the Uniform Convergence of Relative Frequencies of Events to Their Probabilities. Dokl. Akad. Nauk, (4), 1968.
[24] S. Shalev-Shwartz and S. Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, 2014.
[25] J. Maylor and R. C. Williamson. A PAC Analysis of a Bayesian Estimator. Proceedings of the Conference on Computational Learning Theory, 1997.
[26] D. A. McAllester. Some PAC-Bayesian theorems. Machine Learning, 37(3):355-366 1999.
[27] M. Seeger. PAC-Bayesian Generalisation Error Bounds for Gaussian Process Classification. Journal of Machine Learning Research, 3(Oct):233-26,, 2002.
[28] G. K. Dziugaite and D. M. Roy. Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data. Proceedings of the Conference on Uncertainty Artificial Intelligence, 2017.
[29] Langford and J. Shawe-Taylor. PAC-Bayes & margins. Proceedings of the Conference on Advances in Neural Information Processing Systems, 2003.
[30] P. Germain, A. Lacasse, F. Laviolette, and M. Marchand. PAC-Bayesian Learning of Linear Classifiers. In Proceedings of the International Conference on Machine Learning, pages 353- 360. ACM, 2009.
[31] P. L. Bartlett, D. J. Foster, and M. J. Telgarsky. Spectrally-Normaiized Margin Bounds for Neural Networks. In Advances in Neural Information Processing Systems, pages 6240-6249, 2017.
[32] Y. Jiang, B. Neyshabur, H. Mobahi, D. Krishnan, and S. Bengio. Fantastic Generalization Measures and Where to Find Them. Proceedings of the International Conference on Learning Representations, 2020.
[33] M. Pérez-Ortiz, O. Rivasplata, J. Shawe-Taylor, and C. Szepesvári. Tighter Risk Certificates for Neural Networks. arXiv preprint arXiv:2007.1291 2020.
[34] O. Catoni. Statistical Learning Theory and Stochastic Optimization. École d'Été de Proba- bilités de Saint-Flour 2001. Springer, 2004.
[35] O. Catoni. PAC-Bayesian Supervised Classification: The Thermodynamics of Statistical Learning, volume 56 of Lecture notes Monograph Series. Institute of Mathematical Statistics, 2007.
[36] D. McAllester.
PAC-Bayesian Tutorial with Dropout Bound. arXiv preprint
107.21130 2013.
10

---

[37] O. Rivasplata, V. M. Tankasali, and C. Szepesvari. PAC-Bayes with Backprop. arXiv preprint arXiv:1908.07:300 073380 2019.
[38] N. Thiemann, C. Igel, O. Wintenberger, and Seldin. A Strongly Quasiconvex PAC-Bayesian Bound. Machine Learning Research, 76:1-26, 2017.
[39] G. K. Dziugaite and D. M. Roy. Data-dependent PAC-Bayes priors via differential privacy. Proceedings of the Conference on Advances in Newral Information Processing Systems, 2018.
[40] A. Majumdar and M. Goldstein. PAC-Bayes Control: Synthesizing Controllers that Provably Generalize Novel Environments. Proceedings of the Conference on Robot Learning, 87: 293-305, 2018.
[41] S. Veer and A. Majumdar. Probably Approximately Correct Vision-Based Planning using Motion Primitives. Proceedings of the Conference on Robot Learning, 2020.
[42] A. Z. Ren, Veer, and A. Majumdar. Generalization Guarantees for Imitation Learning. Proceedings the Conference Robot Learning, 2020.
[43] A. Majumdar, A. Farid, and A. Sonor. PAC-Bayes control:
learning policies that provably generalize to novel environments. The International Journal of Robotics Research, 40:574- 593, 2021.
[44] Viallard, P. Germain, A. Habrard, and E. Morvant. General Framework for the Deran- domization of PAC-Bayesian Bounds. arXiv preprint arXiv:2102.0866 2021.
[45] D. Wierstra, T. Schaul, T. Glasmachers, Y. Sun, J. Peters, and J. Schmidhuber. Natural evolu- tion strategies. The Journal of Machine Learning Research, 15((7):949 2014.
[46] R. V. Hogg, McKean, and A. Craig. Introduction to Mathematicul Statistics. Pearson Education, 2005.
[47] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. http://pybullet. org, 2018.
[48] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, al. Shapenet: An information-rich 3D model repository. arXiv preprint 1rI2:01512.03012 2015.
[49] A. Maurer. A Note on the PAC Bayesian Theorem. arXiv preprint cs.LG/041 1099, 2004.
[50] M. Gil, F. Alajaji, and T. Linder. Rényi divergence measures for commonly used univariate continuous distributions. Information Sciences, 249:124-131, 2013.


Reviews:
[{"review_id": "aaNqN8rjXro", "reviewer": "Reviewer_ev1V", "summary": "The paper presents a PAC-Bayes approach for out-of-distribution (OOD) detection within policy learning settings. The authors first apply derandomized PAC Bayes to train a policy, which provides a bound per hypothesis rather than averaged analysis over multiples hypothesis. Then, as a bound is provided per hypothesis, the paper proposes to check the violation of the PAC-Bayes guarantees at test time, i.e. if the PAC Bayes bound is violated, the authors attribute this to OOD scenario. Concretely, this is achieved via hypothesis testing and checking the confidence intervals. Theoretically, a bound is established for these two schemes, which holds within a pre-specified interval. The method is task driven, as the expected costs are used. Experiments with 2 simulations and 1 real robot - in particular, the drone performing vision-based obstacle avoidance - the authors validate the approach. Overall, the paper provides an interesting application of PAC-Bayes to OOD detection within the policy learning settings.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n\nPros:\n\n- To my knowledge, the proposed application of PAC-Bayes to OOD detection within the policy learning settings is novel and interesting. Some of the derivation steps that the authors demonstrate, are non-trivial and thought involving.\n\n- The paper's focus on \"task-driven\" OOD detection - only figuring out the OOD samples that affect the performance of the policy - is relevant to safety-critical settings of robot learning, and the paper shows real robot experiments as well as some of the theoretical foundations associated with their method. This is likely to be appreciated at CoRL. \n\nCons\n\n- The paper has several rooms for improving the exposition.\n\nFirst, the margins seem to be somehow reduced, when compare to other CoRL submissions. This can be checked and corrected if inappropriate, during the rebuttal period. Second, while the insights are very clear, it might help to provide some remarks at the end of each theorems - ideally, in a common tongue or insights. Third, for the proofs in the Appendix, it might help to first state the proof paths, and provide the proofs, so that the work is more accessible. For the video, you may want to make it standalone. Currently, it is difficult to get the relevant information before reading the paper. For example, we see the drone crash at OOD settings, but there are no explanations.There are also minor comments which I list at the end of the review.\n\n- The comparisons to the existing approaches are kept minimal when compared to rational works on OOD detection. A compelling comparisons to other OOD methods can further show-case the strength of PAC-Bayes approaches within the considered scenario.\n\nTo back up, the paper compares to MSP - iclr 2017 and MaxLogit - arxiv 2019 (figure 3), which are both targeted for general settings, and it is unclear whether MSP and MaxLogit are the state-of-the-art alternatives. The paper should demonstrate how far the idea has advanced the state-of-the-art OOD detection literature, e.g. compared to other PAC-Bayes OOD detection methods (ref 14-16), as well as other task-driven OOD methods such as ref 18-22.  \n\nThe reason for emphasizing the comparison to the state-of-the-art is that the provided bounds are statistical, and there are chances that the guarantees are violated. An interesting question is then, how meaningful it is to have such guarantees, when the resulting failures are catastrophic.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend accepting this paper. The paper provides principled approach to OOD detection using PAC-Bayes bounds. It seems original, and technical quality is good - the approach is reasonably backed up by the existing theory, and the real robot experiments show-case the success of their method, which is likely to be appreciated at CoRL.\n\nMy main concern is on the clarity of presentation (which is good but still has rooms to improve - the bars are usually high with theory) and lack of rigorous comparisons to the state-of-the-art alternatives.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "- eq (4) §\\hat{S}§ seems undefined.\n\n- eq (2), eq (3), eq (5), eq (6): P is undefined (though standard in PAC-Bayes, may not be for OOD detection communities).\n\n- figure 2 & 3: indicate what is better or worse, or what the data means intuitively.\n\n- other comments on writing above.\n\n- more comparisons to the state-of-the-art.\n\n- A.17 to A.20 and A.27 to A.30 needs more explanations.\n\n- figure 3 missing the plots of varying environmental difficulties and only shows wind conditions variations.\n\n- setting m bigger than 8, how does this influence the derandomized PAC Bayes? At testing, do you need a batch as well?\n\n- The paper trains a policy with derandomized PAC Bayes. Would this affect the performance of the trained policy over the previous PAC-Bayes policies OR even the state-of-the-art methods? How difficult is it to train as well? Also, how tight is the bound in practice? \n\n- In the appendix, the authors refer to different sources. For the researchers working on OOD detection, providing more background materials in the appendix can help in understanding the paper. For the researchers familiar with PAC-Bayes methods, it can help in not resorting to other papers. \n\n- An empirical validation of the proposed statistical guarantees can be meaningful, e.g. histogram analysis?\n\n\n-----------------------------------------------------------------------------------------------------------------------------------------------\n\nPost Rebuttal\n\n-----------------------------------------------------------------------------------------------------------------------------------------------\n\nI thank the authors for the detailed response. I have thoroughly read the comments from other reviewers, the author response and the revised manuscript. I am happy to see that the revised manuscript has improved in its clarity.\n\nHowever, I maintain my score; I find the paper to be a borderline due to the following reasons:\n\n-  (On significance) more informative comparison to the state-of-the-art.\n\nThe provided use-case of the method is clearly on OOD detection for the policy learning, and yet, the advancements to the OOD literature is unclear.\n\nMy first point is on the unclear motivation behind the chosen baselines. As evidenced by Section 2, the authors state that the missing gaps in \"Anomaly/OOD detection in supervised learning\" is the need for the specific network outputs, e.g. softmax (classification) and perhaps covariance (regression). However, the authors still compare to the approaches for supervised learning: MSP and MaxLogit. Therefore, I found the motivation behind the chosen baselines unclear.\n\nMy second point is the need to choose other \"task-driven OOD detection approaches\" as the baselines. As evidenced by Section 2, the authors discuss other \"task-driven OOD detection\" approaches, and point out that the missing gaps are the statistical guarantees. However, the advancements to the literature is unclear as the authors do not provide any comparison to other \"task-driven OOD detection\" approaches. Therefore, I found the advancements to the state-of-the-art unclear.\n\nWhile the authors address my concerns to an extent, I still found the arguments not fully convincing. For example, the argument that MaxLogit (in 2017) has shown to outperform other alternatives is not convincing. This is because the settings of the experiments are different, and the comparisons of MaxLogit are limited to the approaches prior to 2017. Moreover, the reason behind not comparing to other \"task-driven OOD detection\" approaches is because the proposed method requires the \"roll-out\" in some test environments, which is the limitation of the proposed method over the existing methods. In this regard, the readers could appreciate more if the authors have resolved this limitation, and provided a comparison to the other task-driven OOD detection methods.\n\nThis is the reason why I rate the significance of the paper low; the paper doesnt have to outperform the other approaches, but should contain the required information for the readers to illustrate the advancements to the current approaches.\n\n-  (On significance) the required roll-outs in test environments.\n\nTaking more global view, many existing approaches do not require the roll-outs in the test environments, which is the main point for developing OOD detectors (targeting at safety-critical applications). Simple examples are the baselines chosen by the authors - MSP [3] and MaxLogit [5] (and many others, e.g. reference [4, 17]). Therefore, my question still remains: how meaningful it is to have the provided method with some guarantees, when the resulting failures are catastrophic (over the existing task-driven methods despite the given drawback of the approach).\n\n-  (On thoroughness) the analysis of the batch-size at testing.\n\nThe proposed method can be sensitive to the chosen batch-size at testing (as the authors provide the information). Then, perhaps, an ablation study that guides the choice of an optimal batch-size would be meaningful.\n\n-  (On thoroughness) more data samples for varying environment difficulty.\n\nOnly 2 evaluation points for varying environment difficulty might also be limited in thoroughness, as oppose to more data points with wind variations. The paper could improve with more data samples for varying environment difficulty.\n\n\n-----------------------------------------------------------------------------------------------------------------------------------------------\n\nPost Rebuttal\n\n-----------------------------------------------------------------------------------------------------------------------------------------------"}}, {"review_id": "K79EBPcxU8", "reviewer": "Reviewer_PvzK", "summary": "This paper presents a method to detect OOD scenarios for robots. The method takes into account only changes that impact the robot’s performance, meaning that it neglects distribution shifts that do not affect task completion. This detection technique comes with guaranteed confidence bounds on performance, specifically on the expected cost of the policy. This bound is derived based on recently developed results on derandomized PAC-Bayes. The effectiveness of the method is shown on both simulated and real robotic experiments.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\nThe paper is very well written and structured. One can clearly see that the authors put a lot of care into defining everything thoroughly, and making sure that the readers do not have to go back and forth between sections. The experiments are also well designed and clearly organized. The results show an improvement over the baselines, which detect OOD even in cases with near-identical test expected cost (to the expected training cost), proving that the presented OOD detection is task-driven. \nThe task-driven formulation also helps when the robot’s sensor observations do not include possible disturbance factors as wind strength or direction.\n\n### Weaknesses\nAs already pointed out by the authors in the conclusion, when using this pipeline one still incurs the risk to fail on the test environments, which is exactly what one wants to avoid, especially when dealing with autonomous vehicles.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents an elegant way to detect OOD environments and training a deterministic policy with a guaranteed bound on the expected cost. When the bound is violated, it indicates that the robotic system is not able to maintain the same performance - up a threshold - on the test set.\nThey propose two methods which both work better than the presented baselines. The experiments on the drone also show that the environment is not considered OOD until the wind speed is increased to 75% of the maximum disturbance. The main advantage of their method is two fold:\n- not detecting OOD when the cost is unaffected by the changes, so in a way, not overestimating the proportion of OOD environments\n- detecting OOD in scenarios where not task-driven methods would fail, as in the wind disturbance example\n\nFor this reason, for the clarity of the exposition and for the quality of the experiments, I would recommend acceptance.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "To better inform future practitioners, you could give more intuition on the two methods you proposed, as in, when is better to use p-values vs concentration inequalities.\n", "reviewer_expertise": "Poor: Limited knowledge of the area"}, "extra_text": {}}, {"review_id": "BEaeaFheFb", "reviewer": "Reviewer_orAU", "summary": "This paper introduces a PAC-Bayes theory-based out-of-distribution detection (OOD) framework. \nThe paper proposes to train a policy whose related expected cost on the training distribution is bounded. After training the policy, OOD on the test environments can be performed with two different, proposed methods. The paper contributions are two-fold: guaranteed confidence bounds on OOD detection and sensitivity against environmental changes which matter (i.e. which changes the expected cost under the current policy). ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- provided methods have a bound guarantee which is proven mathematically\n- paper is properly written, can be read fluently\n- interesting and convincing experiments\n\nWeaknesses:\n\n- I don't see any main weaknesses\n- Some more information regarding the policy training procedure would be nice (see Issues)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Even though I am not an expert in this field, I enjoyed reading the paper and learned a lot. The paper is clearly written and can be read fluently. The differences to prior work are made clear. The motivation and claims of the paper are proven via thorough mathematical derivations and experiments. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Poor: Limited knowledge of the area"}, "extra_text": {"issues": "One major remark:\n\n- I think the layout of the paper was changed, especially the space between equations and text. This should be addressed.   \n\nI have minor points that might be interesting to know for the reader:\n\n- It would be interesting to know if the sample complexity of the training algorithms change due to optimizing the bound/\"regularized cost\"\n\n- Theorem 1 requires the training algorithm to be deterministic, as stated in the Appendix. How does this influence the training procedure? Does this mean that usage of stochastic methods is restricted? \n\n- What exactly is meant by transforming the training procedure into a supervised learning setting for the drone experiment?"}}]

Author rebuttal:

Meta-review: The paper proposes a new technique for detecting when the robot is operating in an environment that is different from the one it trained in. This problem is known as out-of-distribution (OOD) detection, and it occurs often in robot learning, especially when new elements (such as wind) are introduced, or when a system is trained in simulation and transferred to the real world. Interestingly, the proposed method does not detect changes that do not affect the performance of the robot. The central idea behind this work is to train the robot with PAC-Bayes performance guarantees. When the robot operates in a given environment, it continuously measures its own performance and compares it against the predicted theoretical performance. When the the difference between the two is sufficiently large, based on p-values of hypothesis testing, then an OOD is detected. The proposed method is evaluated on 2 tasks in simulation and one real robotic task.
The reviewers agree that the technical contribution of this work is solid and novel. The paper is over all well written, although several technical details are not clear. It seems like the paper is written while assuming that the readers are all familiar with the PAC-Bayes literature and notations. Another issue raised by a reviewer is the utility of this approach in practice. It seems like OOD detection after a failure occurs is too late for sensitive robotic applications, such as self-driving cars. In these application, OOD should be tackled before a drop in the performance of the robot, contrary to the central idea of the proposed method here.
Decision: Accept (Poster)