Perceive With Confidence: Statistical Safety Assurances for Navigation with Learning-Based Perception

Authors: anushri  dixit
zhiting  mei
meghan  booker
mariko  storey-matsutani
allen z ren
anirudha  majumdar
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 06:36:38

Abstract: Rapid advances in perception have enabled large pre-trained models to be used out of the box for transforming high-dimensional, noisy, and partial observations of the world into rich occupancy representations. However, the reliability of these models and consequently their safe integration onto robots remains unknown when deployed in environments unseen during training. In this work, we address this challenge by rigorously quantifying the uncertainty of pre-trained perception systems for object detection via a novel calibration technique based on conformal prediction. Crucially, this procedure guarantees robustness to distribution shifts in states when perceptual outputs are used in conjunction with a planner. As a result, the calibrated perception system can be used in combination with any safe planner to provide an end-to-end statistical assurance on safety in unseen environments. We evaluate the resulting approach, Perceive with Confidence (PwC), with experiments in simulation and on hardware where a quadruped robot navigates through previously unseen indoor, static environments. These experiments validate the safety assurances for obstacle avoidance provided by PwC and demonstrate up to 40% improvements in empirical safety compared to baselines.

Main paper: 
 introduction: 
 How can we decide if the outputs of a given perception system are sufficiently reliable for safetycritical robotic tasks such as autonomous navigation? Significant strides in perception over the past few years have enabled large pre-trained models to be used out of the box [1] for tasks such as occupancy prediction, which serves as a fundamental building block for navigation. However, current pre-trained models are still not reliable enough for safe integration into many real-world robotic systems. Despite being trained on vast amounts of data, these systems can often fail to generalize to novel environments [2, 3, 4] . In this paper, we ask: how can we leverage the power of large pre-trained occupancy prediction models while providing safety assurances for robot navigation?Consider a legged robot tasked with navigating in a cluttered environment such as a home, office, or warehouse (Figure 1 ). A typical navigation pipeline for such a system consists of two modules: (i) a perception module that detects obstacles, and (ii) a planner that produces collision-free trajectories assuming accurate perception. However, there are two challenges associated with obtaining reliable outputs from the perception module. First, the environments in which we deploy our robots will be unseen during training, and thus require generalization to new obstacle geometries, appearances, and other environmental factors. Second, closed-loop deployment of the perception system in conjunction with a planner causes a shift in the distribution of states (e.g., relative locations to obstacles) that are visited by the robot. Since the robot's planner influences future states, the robot may view obstacles from unfamiliar relative poses (Figure 1 ) and cause the perception system to fail.In this paper, we address these challenges by performing rigorous uncertainty quantification for the outputs of a pre-trained perception system in order to achieve reliably safe (i.e., collision-free) navigation. We utilize techniques from conformal prediction [5] in order to lightly process the outputs of a pre-trained obstacle detection system in a way that provides a formal assurance on correctness: with a user-specified probability 1 -ϵ, the processed perceptual outputs will correctly detect obstacles in a new environment. To enable this, we assume access to a modest-sized (e.g., |•| = 8th Conference on Robot Learning (CoRL 2024), Munich, Germany.Figure 1 : PWC lightly processes the outputs of a pre-trained perception system (green bounding boxes) using conformal prediction in order to ensure a bounded misdetection rate despite any distribution shift in states (gray dots). The calibrated perception system (blue boxes) paired with a non-deterministic filter and a safe planner provide an end-to-end statistical assurance on safety in new test environments. 400) dataset of environments that are representative of deployment environments with ground-truth obstacle annotations, and use these for calibrating the outputs of the perception system. Crucially, we propose a novel calibration technique that ensures robustness of the perception system to any closed-loop distribution shift in states. Hence, the calibrated outputs can be used in conjunction with any safe planner to provide an end-to-end statistical assurance on safety in new static environments with a user-specified threshold 1 -ϵ. To the best of our knowledge, this is the first work to calibrate a given black-box perception system in a way that ensures robustness to closed-loop distribution shifts in order to provide end-to-end statistical assurances on safe navigation.Our framework, Perceive with Confidence (PWC), is evaluated with experiments in simulation and hardware on the Unitree Go1 quadruped navigating in indoor environments with objects that are unseen during calibration (Figure 1 ). We validate PWC's ability to provide end-to-end statistical assurances on collision avoidance, while also providing up to 40% increase in safety with only modest reductions in task completion rates compared to baselines that use the pre-trained perception model directly, fine-tune it on the calibration dataset, or utilize conformal prediction for uncertainty quantification but do not account for closed-loop distribution shift. 
problem formulation and overview: 
 Dynamics and environments. Suppose that the dynamics of the robot are described by s t+1 = f E (s t , a t ), where s t ∈ S is the robot's state at time-step t, a t ∈ A is the action, and E ∈ E is the environment that the robot operates in during a given episode. We primarily focus on navigation with static obstacles; in this context, the environment E specifies the locations and geometries of objects. We assume that environments that the robot will be deployed in are drawn from an unknown distribution D E , e.g., a distribution over possible rooms that the robot may be deployed in. We will make no assumptions on this distribution besides the ability to sample a finite datasetD = {E 1 , . . . , E N } of i.i.d. environments from D E .Sensor and perception system. The robot is equipped with a sensor σ : S × E → O that provides observations o t = σ(s t , E) (e.g., depth images) based on the robot's state and environment. We assume access to a pre-trained perception model ϕ : O → Z, which processes raw sensor observations into an occupancy representation of the environment. In this paper, we work with perception models for obstacle detection that output 3D bounding boxes. The representations (z 0 , . . . , z t ) up to the current time-step are aggregated into an overall representation m t ∈ M (e.g., a map).Policy. The representation m t is used by a planning algorithm in order to produce actions. Denote the resulting end-to-end policy that utilizes a perception model ϕ by π ϕ : O t+1 → Z t+1 → M → A, which maps histories of sensor observations to actions.Safety and task performance. Let C safe E be a cost function that captures safety (e.g., obstacle avoidance). Specifically, let S 0,E denote the allowable set of initial conditions in environment E. Then, C safe E (π ϕ ) ∈ {0, 1} assigns a cost of 0 if policy π ϕ maintains safety from any initial state s 0 ∈ S 0,E when deployed over a given time horizon in environment E, and a cost of 1 otherwise. An additional cost function C task E can be used to capture task performance (e.g., time to reach a goal).Goal: statistical safety assurance. Our goal is to provide a statistical assurance on safety for the end-to-end policy π ϕ . We propose a procedure that uses a finite dataset D of environments in order to produce a calibrated perception system φ :O ϕ -→ Z ρ -→ Z.Our approach is modular: outputs of the calibrated perception system may be used with any safe planner (cf. Section 5) to ensure:EQUATIONfor a user-specified safety tolerance ϵ, while also post-processing outputs from ϕ as lightly (i.e., non-conservatively) as possible in order to allow the robot to optimize task performance. 
background: conformal prediction: 
 Conformal prediction (CP) [5, 6] will be our primary tool for performing rigorous uncertainty quantification for perception. Given N i.i.d. (or exchangeable) samples U 1 , . . . , U N of a scalar random variable U , we compute the threshold, q1-ϵ , such that the next sample, U test , satisfies,EQUATIONwhere U (1) ≤ U (2) ≤ . . . ≤ U (N )EQUATIONwhere, Beta -1 N +1-v,v (δ) is the δ-quantile of the Beta distribution with parameters N + 1 -v and v, and we can choose ε to achieve the desired 1 -ϵ coverage. 
offline: calibrating the perception system: 
 In this section, we describe our approach to the uncertainty quantification of a pre-trained perception system. We focus on the challenges highlighted in Section 1: providing statistical assurances on safe generalization to novel environments and ensuring that the offline calibration procedure is robust to shifts in the distribution of states induced by the online implementation of the planner.We focus on perception systems that output bounding boxes that predict the locations of objects in the environment. As an example, Figure 1 (left) shows one such real-world environment wherein the union A of the black boxes denotes the ground-truth locations of the chairs. Let B s denote the union of the green bounding boxes predicted by the perception system ϕ from robot state s ∈ S. Since the environment in which the robot is deployed may contain partially occluded objects that ϕ was not explicitly trained on, the perception system's outputs may be inaccurate.Our key idea for ensuring generalization to new, unseen environments and tackling the distribution shift arising from the closed-loop deployment of the calibrated perception system with a plannner is to use a policy-independent misdetection cost, CE , which considers worst-case errors across all states in an environment 1 , CE (ϕ) := max s∈S 1 A̸ ⊆Bs . We will present a calibration procedure that bounds this misdetection cost with high probability in a new environment, and thus guarantee the correctness of the calibrated perception system independent of the robot policy using CP.Dataset. We assume access to a dataset of N i.i.d. environments D = {E 1 , . . . E N } ∼ D E (cf. Section 2). In each environment, E i , we have access to the union A i of the ground-truth bounding boxes of all the objects in the environment and the unions B s,i of the predicted bounding boxes generated by the pre-trained perception system ϕ from each state s ∈ S. Care is required to ensure that the calibration environments are representative of deployment environments. As such, we construct the calibration dataset either using real-world environments or create simulation environments using real-world data [9, 10, 11] to ensure sufficient variation in environmental factors (e.g., geometry and locations of obstacles, lighting, etc.).Calibration. In each calibration environment E i , we find the inflation ∆ qi of the bounding box predictions B s,i so as to ensure that all the ground-truth boxes are fully enclosed by the inflated boxes, i.e, A ⊆ B s,i +∆ qi , ∀s ∈ S, where, S is assumed to be a finite, discrete set. Here, B s,i +∆ qi is the inflation of each bounding box in the union B s,i by 2q i along each dimension. We define the non-conformity score for environment E i to be the minimum required inflation in that environment:EQUATIONObserve that U i ≤ 0 =⇒ A i ⊆ B s,i , ∀s ∈ S and a growing U i signals a worse performance of the pre-trained perception system. We can compute the nonconformity scores for the i.i.d. sampled environments {E 1 , . . . , E N } and the quantile q1-ϵ = Quantile U (1) , . . . , U (N ) ;⌈(N +1)(1-ε)⌉ N 2 .Proposition 1 Consider the calibrated perception system φ that modifies every bounding box output of the perception system ϕ by scaling the predicted bounding boxes as B = B + ∆ q1-ϵ . With probability 1 -δ over the sampling of the dataset used for calibration, the calibrated perception system, φ, is guaranteed to have an ϵ-bounded misdetection rate on new test environments:EQUATIONThe above proposition (proof in Appendix A) gives us a formal assurance on the correctness of the perception system independent of the robot's policy. As we describe below, the calibrated perception can thus be combined with any safe planner to bound the collision rate to ϵ. We now focus on the online implementation of the method described in Section 4 to reduce conservatism when used in conjunction with a safe planner. In general, a safe planner takes into account the dynamics of the robot and produces plans in the state space S. We call X the configuration space of the robot (e.g., x-y location for a point). For any given environment E, we partition X into three sub-spaces: the known free space X free , known occupied space X occ , and unknown space X unknown .Non-deterministic filter. We utilize the assurance obtained from Section 4 to implement a nondeterministic filter [12, Ch. 11.2.2] which shrinks the occupied space and grows the known free space over time. Suppose the robot's perceived partition of the state space X at time t is denoted by the triplet{X free t , X occ t , X unknown t}, which represents the overall map m t of the environment. At a new time step t + 1, the robot's perception system returns a new estimation for the occupied space, X occ t+1 . The filter intersects the occupied spaces:X occ t+1 = X occ t ∩ X occ t+1 .We compute the new estimation of free space X free t+1 based on X occ t+1 , considering occlusion and limited field of view.The new perceived free space is updated by taking the union:X free t+1 = X free t ∪ X free t+1 .The nondeterministic filter pairs effectively with our method in Section 4 for two key reasons: 1) it mitigates the conservatism of our bounding box expansion by intersecting X occ t , rapidly reducing its size even if the initial prediction with CP bounds appears generous; and 2) Prop. 1 ensures that with high probability in a new test environment, X free t never intersects the true occupied space X occ . We demonstrate the rapid expansion of known free space in Figure 3 for our simulated setup (Sec. 6).Safe planning. With our formal assurance on the estimated free space X free t , we can utilize any safe planner [13, 14, 15] to ensure end-to-end safety, as long as the planner includes a safety filter that takes into account the robot's dynamics in order to reject potentially unsafe actions with the assumption of known state and static (but unknown) environment [16, Corollary 1.4 ]. For our simulation and hardware experiments, we use the safe planner proposed in [17] , which enforces an inevitable collision set (ICS) constraint [18] . We describe implementation details in Appendix C.Proposition 2 For any user-specified safety tolerance ϵ, the calibrated perception system φ in Proposition 1 combined with any safe planner that chooses actions based on the outputs of the non-deterministic filter ensures the end-to-end safety for the overall policy π φ:EQUATIONwhere C safe E (π φ) is the cost function for safety from Section 2.This result (proved in Appendix D) is a direct consequence of the formal assurance on the calibrated perception system that ensures correctness from any state in a new test environment (sampled i.i.d. from the same distribution as the calibration environments) with probability 1-ϵ over environments.6 Simulated Experiments: Vision-Based Navigation The robot begins with large occupied space predictions due to the inflation obtained through offline calibration (Section 4). After a few updates, the predicted occupied space X occ shrinks significantly.We evaluate our approach for vision-based navigation in the PyBullet simulator [19] using a diverse set of chairs from the 3D-Front dataset [11] . We use the 3DETR end-to-end transformer model [20] as our pre-trained perception system.Baselines. We compare our approach (Perceive with Confidence -PWC) to three baselines to illustrate its effectiveness in achieving a user-specified safety rate. First, we consider the most common approach of directly using the outputs of the perception system [20] in our planning pipeline. We call this baseline 3DETR. Next, we consider the common practice of fine-tuning the outputs of the perception system using a small dataset of task-representative environments D tune (cf. Section E.1). We call this perception system 3DETR-fine-tuned. Lastly, we perform calibration using conformal prediction; however, instead of accounting for the closed-loop distribution shift, we bound the (2) misdetection rate averaged across environments and states (similar to [21] , which does not utilize conformal prediction, but quantifies expected errors in a perception system for a pre-defined distribution of states). We refer to this baseline as CP-avg. We consider two variations of our approach for comparison to the above baselines. First, we refine 3DETR outputs using our calibration procedure described in Section 4. We call this approach PWC. Second, the 3DETR outputs are fine-tuned and calibrated using split conformal prediction as described in Appendix E.1; we call this approach PWC-fine-tuned. Details regarding calibration and the planner setup are provided in Appendix F. Results: Misdetection Rate. We first compare our method, PWC, to the baseline CP-avg that is also calibrated using conformal prediction but without accounting for the closed-loop distribution shift. We compare the misdetection rate, i.e., whether obstacles in the scene are classified as free space at any point during a trial. We vary the allowable misdetection bound ϵ for each method, and compute the rate of misdetections in 100 test environments. As seen in Figure 5 , our method is guarantees a rate of misdetection lower than the threshold ϵ while CP-avg violates this threshold for every ϵ considered.3, 0] , 4, 0] [1, 3, 0, 4]Results: Collision Rate. We compare PWC to the baselines in 100 new environments drawn from the same distribution as calibration environments. Figure 3 illustrates one such test environment and the evolution of the free space in this environment using PWC. Figure 3 shows that though the initial calibrated perception system outputs are inflated, the non-deterministic filter is able to expand the predicted free space in a few time steps and ensure that the robot can navigate without unnecessary conservatism. The results are summarized in Figure 4 and the metrics for success and failure are described in Appendix F. We observe that our proposed approaches, PWC and PWC-finetuned, have no collisions in any environments. While the robot reaches the goal in a slightly lower percentage of environments compared to baselines, we emphasize that ours is the only approach that is able to ensure a low, statistically guaranteed misdetection rate across test environments. To further illustrate the effect of misdetections on safety, we consider a different distribution of environments wherein we randomly place a single chair in the straight line path between the initial position of the robot and the goal. For a safety threshold 1 -ϵ = 0.85, we compare PWC, CP-avg, and 3DETR. The results are provided in Figure 6 for 100 new test environments, wherein the goal is reached if the robot navigates to within 2 m of the goal. In these environments, the desired safety rate is not met by the baselines while our approach is still statistically guaranteed to be safe.We provide additional simulation results to illustrate the effects of 1) closed-loop distribution shifts on safety in Appendix F.4 wherein PWC is robust to an increase in the level of closed-loop distribution shift while the baseline, CP-avg., is not which leads to higher collision rates for CP-avg., 2) the tradeoff in different partition sizes for fine-tuning using split-CP in Appendix E.1.2, 3) the effect of varying ϵ on the safety rate in Appendix F.2, 4) impact of using different number of sampled configurations for calibration and online planning in Appendix F.3, and 5) comparison against additional uncertainty-aware perception systems that use a heuristic notion of uncertainty in Appendix F.5. 
hardware validation: vision-based quadruped navigation: 
 We now validate the end-to-end statistical safety assurance of our approach on a quadrupedal hardware platform. As in our simulation setup in Section 6, the robot is tasked with navigating to a goal location while avoiding different chairs placed in varying configurations across a 8m x 8m room. We utilize the perception system calibrated in simulation with a guaranteed safety rate of 1 -ϵ = 0.85, and compare our PWC method against CP-avg. (defined in Section 6) across 30 different physical environments (60 trials total). One challenge is to ensure a minimal sim-to-real gap for perception.In order to address this, we utilize depth measurements as the robot's sensory input. This choice facilitates a small sim-to-real gap, as observed in prior work [22, 23] . See Appendix G for more details about the hardware setup. Results. For PWC, we used the q0.85 = 0.73m threshold found in simulation to inflate the predicted bounding boxes returned from 3DETR in order to achieve 85% confidence that our robot will remain safe in new environments. We summarize key statistics of PWC compared against CPavg. (q 0.85 = 0.02) across 30 different environments in Figure 4 (right). Importantly, our trials demonstrate that our confidence bound holds on hardware in real environments and without being too conservative. PWC was safe through 90% of the trials and also had comparable path length to the baseline. Meanwhile, the baseline struggled in the real environments by having misdetections in each trial and colliding with a chair in half of the trials. See Figure 7 for trajectories and free space estimations through several environments with narrow spaces, occluded chairs, and occluded goals.The supplementary video contains full example trials.PWC's low misdetection rate and higher success rate in these trials emphasize the efficacy of the bounding box inflation provided by CP paired with the non-deterministic filter. This pairing, in a principled way, inflates the (potentially poor) bounding box detections to properly capture obstacles but quickly shrinks the occupied space with the filter such that the robot can still navigate effectively. 
related work: 
 Safe planning. Collision avoidance is a crucial goal in autonomous navigation. Safe planning methods typically rely on the assumption that the robot has perfect knowledge of its state and environment [16] . Recent approaches have allowed for occlusion [17, 24, 25, 26] or accounted for losing sight of a previously tracked object [27] , but still require either perfect detection of seen objects or bounded sensor noise. Such assumptions are impractical for learning-based perception modules that can fail catastrophically in new environments.Formal assurances for perception-based control. Proposed methods include control barrier functions (CBFs) [28, 29] , verification methods on neural networks (NNs) [30, 31] , and other learningbased methods [31, 32, 33, 21, 34, 35, 36, 37] . However, these works either do not guarantee generalization to novel environments [30, 31] , or ignore closed-loop distribution shifts [34, 21] , or require end-to-end training and a good prior [35, 36, 37] , or demand usage/design of specific controllers [28, 29, 32, 38] . Some make strong assumptions on the perception system [39, 40] that are unrealistic for deployment. In contrast, our method doesn't need any of the above, and is lightweight and modular, allowing for the use of any downstream safe planners to ensure end-to-end safety.Conformal prediction. Conformal prediction (CP) [5, 7, 8] is an uncertainty quantification framework particularly suitable for robotics applications [41, 42, 43, 44] where learned modules are deployed in environments drawn form unknown distributions. In this work, we focus on providing uncertainty quantification for the perception system, which usually involves high-dimensional inputs and closed-loop distribution shifts. Prior works [43, 21, 45, 46] either provide guarantees for a single environment, assume known environments, or do not account for closed-loop distribution shifts. To the best of our knowledge, this is the first work to obtain end-to-end safety assurances for the perception and planning system in new environments while being robust to closed-loop distribution shifts and amenable to changes in the planner parameters. 
discussion and conclusions: 
 We presented a modular framework for rigorously quantifying the uncertainty of a pre-trained perception model in order to provide an end-to-end statistical safety assurance for perception-based navigation tasks. Notably, our statistical assurance holds for generalization to new environmental factors (e.g, new obstacle geometries and configurations) and allows for the distribution shift of states that may occur during closed-loop deployment of the perception system with the planner. Our simulation and hardware experiments validated the theoretical safety assurances provided by PWC, while demonstrating significant empirical improvements in safety compared to baseline approaches that do not consider closed-loop distribution shift.Limitations and future work. One limitation of our work is the assumption of static obstacles. As a future direction, we are interested in quantifying uncertainty in both the state of agents moving in the environment and predictions of their semantic labels (e.g., "pedestrian" vs. "bicyclist"), and utilizing game-theoretic planning techniques that account for the uncertainty in the agents' current state and future motion. Additionally, the inflation of bounding boxes we acquire from CP introduces some conservatism. We outline an extension to our approach in Appendix E to address this challenge by utilizing more general occupancy representations beyond bounding boxes, e.g., scene completion networks [47] , which produce voxel-wise occupancy confidences. Constructing different non-conformity score functions that incorporate confidences from a pre-trained model could also potentially reduce conservatism. Lastly, we are interested in uncertainty quantification for perception models that support tasks beyond point-to-point navigation, e.g., calibrating the outputs of multi-modal foundation models for language-instructed navigation where we ensure accurate detection. We expect that rigorous uncertainty quantification is a necessary step towards fully leveraging the power of large foundation models [1] while safely integrating them into future robotic systems. 
 
 Appendix: As seen in Appendix 3, conformal prediction gives us the following dataset-conditional guarantee on a new sample of the nonconformity score U test corresponding to a test environment E test . With probability 1 -δ over the sampling of U 1 , . . . , U N ,P[U test ≤ q1-ϵ |U 1 , . . . , U N ] ≥ Beta -1 N +1-v,v (δ).We can rewrite the event U test ≤ q1-ϵ as:{U test ≤ q1-ϵ } = q1-ϵ ≥ min qtest q test |A test ⊆ B s,test + ∆ qtest , ∀s ∈ S = A test ⊆ B s,test + ∆ q1-ϵ , ∀s ∈ S ={A test ⊆ Bs,test , ∀s ∈ S} = CEtest ( φ) = 0 ,which gives us the desired result (5).A natural question that arises after following the calibration procedure described above is: what happens if the robot is not able to observe all objects in the environment from all states? This may happen due to a limited sensing capability or because some parts of the environment are occluded from view. We address this issue in our calibration procedure implementation by only taking into account perception errors for objects that are within the field-of-view of the robot in a given state, and masking any ground-truth bounding boxes that are not visible to the robot, i.e., A (which now depends on state s) is the union of all the ground-truth bounding boxes of the visible objects. Hence, the perception system correctness assurance stated above holds for all objects within the field-ofview of the robot at any given state. The presence of possibly occluded obstacles are dealt with by a safe planner, which we describe next.For our simulation and hardware experiments, we use the safe planner proposed in [17] due to its approximate optimality and ease of implementation. The safety filter in this case is an inevitable collision set (ICS) constraint [18] , where the robot is forbidden to enter any state that will eventually result in collision no matter what control actions are taken. Within the known free space X free t , the robot plans using the fast marching tree algorithm (FMT ⋆ ) [48] with dynamics [49] . If the goal is not visible within X free t , the robot plans to an intermediate goal on the boundary of its free space. The intermediate goals are chosen based on the cost-to-come from current robot state to the intermediate goal, and the distance-to-go from the intermediate goal to the actual goal. The robot replans whenever it receives a sensor update and an updated X free t+1 from its non-deterministic filter, and accounts for ICS constraints [50] in-between sensor updates.As shown in Proposition 1, the misdetection rate of the calibrated perception system φ is ϵ-bounded on environments drawn from D at each time step t, where the robot is at state s t . In other words, the predicted occupied space X occ t at each time step contains the true obstacles A with high probability across environments. Conversely, the predicted free space X free t at each time step does not contain the true obstacles A with high probability across environments. If we consider a safety-relevant misdetection cost at time step t:EQUATIONthen the misdetection rate over the set of states should be ϵ-bounded across environments by Proposition 1:EQUATION)Because the expectation in Equation ( 8) is over the set of environments, the following statement holds in any new environment (with probability 1 -δ over the calibration dataset of environments),EQUATION)Given m t = {X free , X occ , X unknown }, a safe planner never drives the robot outside of the free space.Therefore, the safe planner guaranteesEQUATIONIn this section, we outline a few extensions to the basic technical approach described in Sections 4 and 5: (i) fine-tuning a pre-trained perception model, (ii) incorporating sensor and dynamics uncertainty, and (iii) calibrating perception modules beyond bounding box prediction.In Section 4, we assumed access to a pre-trained perception model ϕ that outputs bounding boxes. The conformal prediction-based uncertainty quantification procedure then uses the calibration dataset D = {E 1 , . . . , E N } of environments to produce a calibrated perception system φ which lightly processes the outputs of ϕ by inflating the predicted bounding boxes. In practice, it may also be useful to fine-tune ϕ for our target deployment environments before performing uncertainty quantification.This can be achieved using split conformal prediction [8] , where one splits the overall dataset D into D = D tune ∪ D cal . If the perception model takes the form of a neural network ϕ w parameterized by weights w, we can use D tune to fine-tune w (or the weights of a residual network). We can then utilize D cal in order to perform the CP-based calibration as described in Section 4. As we demonstrate in Section 6, this additional fine-tuning step before calibration can reduce the conservatism of outputs and improve end-to-end success rates.The typical choice of loss function for training a bounding box predictor is the generalized intersection-over-union (gIoU) loss [51] . This is a differentiable version of the IoU loss: given a ground-truth bounding box A and a predicted box B, one computes L(A, B) := |A ∩ B|/|A ∪ B|. However, while this loss is popular in computer vision, it is not suitable for robot navigation. In particular, the IoU loss is symmetric: it does not distinguish between the ground-truth and predicted bounding box and thus does not encourage the predicted box to contain the ground-truth box. We propose a modification to the gIoU loss in Appendix E.1.1, which encourages that the predicted bounding box encloses the ground-truth box while also ensuring that the predicted box is not too large. Similar to the gIoU loss, this loss is (almost-everywhere) differentiable and scale invariant. We utilize this loss for fine-tuning in our experiments (Section 6). However, one could use any other method for finetuning not limited to training a simple neural network with gIoU loss [52] . We use a (almost-everywhere) differentiable loss function for training. The loss function seeks to ensure that the predicted shape (e.g., bounding box) encloses the ground truth shape while also ensuring that the predicted shape is not too large.Let's consider the simplest setting wherein we have one object in the scene and we are making a single prediction. In this case, A denotes the (convex) ground-truth shape and B denotes the (convex) predicted shape. Let C denote the convex hull of A and B. Our loss function is a weighted combination of three terms,L := w 1 l 1 + w 2 l 2 + w 3 l 3 = w 1 |A\B| |A| + w 2 |B\A| |B| + w 3 |C\(A ∪ B)| |C| .The first term is the most important; it tries to ensure that B encloses A. The second term tries to make sure that B is not much larger than it needs to be, see Figure 8 . The first and second terms are sufficient if A and B are overlapping. However, if they do not overlap, there is no gradient information provided by the first two terms. Following [51] , we introduce a third loss term in order to provide gradient information when the shapes do not intersect. The loss terms l 1 , l 2 , l 3 are each bounded within [0, 1]. Hence, if we choose w 1 , w 2 , w 3 such that i w i = 1, then the overall loss is also bounded within [0, 1]. Now let's consider the setting wherein, A denotes the union of multiple ground-truth bounding boxes (say we have m objects in the scene) and B is the union of all the predicted bounding boxes (we predict n boxes). We consider all the individual bounding box predictions B i , ∀i ∈ {1, . . . n} and associate the closest visible ground-truth bounding box A i to each prediction. Now we can define C i as the convex hull of A i and B i and the resulting loss function, L i ,L i := w 1 |A i \B i | |A i | + w 2 |B i \A i | |B i | + w 3 |C i \(A i ∪ B i )| |C i | .Hence, the overall loss is,L = 1 n n i=1 L i .Please refer to [51, Appendix 4.3] for instructions on how to compute the loss analytically for axis-aligned bounding boxes.Upon collecting a calibration dataset of ∼ 400 environments, as described in the experiment setup in Section 6, we may choose to use a smaller subset of the calibration dataset to further finetune the pre-trained perception model to perform better in the types of environments we are interested in deploying the robot in. We consider the effect of different dataset split sizes for finetuning and then calibration. Using a larger set of environments for finetuning |D tune | may result in a better tuned model, but will leave fewer environments for calibration, |D cal |, resulting in a more conservative ε and q1-ϵ that satisfies the dataset-conditional guarantee (3), and vice versa. This trade-off is seen in Table 1 , where we observe the best performance when we have an equal split between finetuning and calibration.Split In Section 2, we modeled the robot's sensor as a deterministic mapping σ : S × E → O, which provides observations from a particular state in a given environment. This formulation allows us to also incorporate sensor errors. Specifically, any errors or randomness in the sensor can be formally included as part of the environment E ∈ E. Thus, in addition to sampling environmental variables such as obstacle locations, geometries, etc., each environment E also samples random variables that prescribe sensor errors from each state s ∈ S in the environment. This way of modeling sensor errors allows σ to be deterministic (since all sources of randomness are included in E), allows the sensor errors to be dependent on the relative pose of the robot relative to obstacles (e.g., modeling the fact that depth estimates are often further from ground-truth depth values as distance increases), and also allows us to model correlations in sensor errors from different locations (e.g., capturing the fact that sensor errors from nearby robot locations can be highly correlated). Modeling time-varying sensor errors (i.e., different sensor errors from the robot state at different times) is not as immediate, but could potentially be incorporated by augmenting the state space S to include the time-step.In addition to errors in sensing, one can also account for uncertainty in the dynamics of the robot by using a robust planner (see [16] for an overview). In the experiments described in Section 7, we incorporate uncertainty by generating plans that prevent the robot from entering the inevitable collision set (cf. Section 5) even with bounded uncertainty in the dynamics.Section 4 introduced the CP-based calibration procedure in the context of bounding box prediction. However, the theoretical formulation in Section 4 is applicable to more general occupancy prediction models; the key requirement is the presence of a scalar quantity that monotonically grows the size of the predicted occupied space (e.g., the inflation parameter q for bounding boxes in Section 4). This allows one to define the non-conformity score U i for an environment E i as in (4) to be the smallest scalar such that the inflated predicted occupied space contains the ground-truth obstacles (for all robot locations). Hence, we can calibrate the outputs of any perception system that predicts an occupied set or performs occupancy prediction more generally, i.e., assigns a (heuristic) occupancy confidence to each point in the space. Possibilities for the latter include scene completion networks [47] or deep signed-distance function representations [53] . A threshold on this confidence acts as the scalar parameter that monotonically controls the size of the predicted occupied space. The conformal prediction procedure from Section 4 can then be used to find a confidence threshold such that predicted occupied space contains the true occupied space (with probability 1 -ϵ in a new environment).We collect a calibration dataset of 400 environments wherein we randomly place 1 -5 chairs from the diverse 3D-Front dataset [11] in a 8 m ×8 m room. In this 8 m ×8 m space, we use a fixed set of 2000 sampled configurations for the sampling-based motion planner and use the same set of samples for the calibration procedure. We construct the calibration dataset in simulation using CAD models of real furniture pieces from the 3D-Front dataset [11] , which contains a highly diverse array of industrial CAD models developed by professional designers to ensure that the performance of the perception system remains the same in its simulation and hardware implementation. Similarly, we collect an additional fine-tuning dataset D tune consisting of 100 environments. These environments include ones with occlusions of the goal and objects in the scene.We simulate the dynamics of the Unitree Go1 quadruped robot and task the robot with navigating to a goal location that is ∼ 7m away from the initial location of the robot. The robot camera has a field of view of 70 • and a visibility range of [1, 5] m. With an allowable misdetection rate of ϵ = 0.15, we obtain q0.85 = 0.75 m for PWC, q0.85 = 0.65 m for PWC-fine-tuned, and q0.85 = 0.05 m for CP-avg. through calibration. The planner replans and obtains a new sensor observation to update the filter every 0.5 s or less (if the previous plan is already completed).We utilize the following metrics for our simulation experiments: a trial is counted as a collision if the robot collides with an obstacle and we count a misdetection for a trial if the free space predicted by the planner has any intersection with the ground-truth bounding boxes of the obstacles. We say that the goal has been reached in a given trial if the robot is able to navigate to within 1 m around the goal in less than 140 s. We also record the average path length for trials in which the goal is reached. We compare our method, PWC, to the baseline CP-avg. We vary the allowable safety rate ϵ for each method, and compute the rate of safety in 100 test environments. As seen in Table 2 and Figure 5 , our method guarantees not only that the rate of misdetections are guaranteed to be bounded, but also the safety rate. The safety rate of PWC is also consistently better than that of CP-avg. For our experiments, we used a fixed set of 2000 sampled configurations. However, depending on the planner configuration requirements and desired speed of computation, the user may decide to have a different number of configuration samples for calibration and planning. We study the change in the CP inflation, q0.85 , the resulting collision, misdetection, and task completion (reaching goal) rates. As we can see in Table 3 , in our case, we have far fewer misdetections with fewer samples, but we also observe a decrease in number of times the robot reaches the goal. We suspect that with fewer samples of configurations (consisting of x, y, v x , v y ), it is harder for the sampling-based motion planner to find feasible paths. On the other hand, we also observe a less conservative q0.85 when we use fewer samples; this is presumably also a result of using fewer samples to compute the non-conformity score that comprises of the worst-case perception error across all configurations. In addition to the challenge of generalization, we highlight another challenge that any uncertainty quantification method for perception must tackle. Suppose we fix a policy π ϕ (that uses perception system ϕ) and collect a dataset of observations in different calibration environments from the states that result from applying π ϕ . We can use groundtruth bounding boxes in these environments to produce a calibrated perception system φ with a statistical assurance on correctness for the distribution of observations induced by π ϕ . However, if we now apply the policy π φ using the calibrated perception system φ, the resulting distribution of states will be different from the distribution that forms the calibration dataset, thus invalidating the statistical assurance. We refer to this challenge as closed-loop distribution shift, which is similar to challenges that arise in offline reinforcement learning [54] and imitation learning [55] .F.To illustrate the effect of closed-loop distribution shifts on misdetections, we used exactly the same setup described above to obtain the simulation results in Figure 4 . We changed the planner cost to have a different weighting on the cost-to-go. For one setting, we chose a weight w = 1 on the cost-to-go, which is the same as the weighting on the cost-to-come. In another setting, we chose a weight w = 10 on the cost-to-go, and hence a 10× more emphasis on the cost-to-go compared to the cost-to-come. Table 4 shows the KL-divergence between the states visited by the planner and the sampling distribution of states for calibration as a measure of the closed-loop distribution shift. Increasing closed-loop shifts lead to higher misdetections. One can see that a simple change in the planner parameters can lead to potentially large changes in the safety rates for CP-avg. The closedloop shift we may see in practice is unknown apriori. Hence, it is difficult to make any statements on the planner safety in closed-loop despite using CP for calibration of the perception system. PWC, on the other hand, is robust to the closed-loop shifts and can still satisfy the misdetection and safety assurance regardless of the planner parameters used. We compare PWC to the baseline method of inflating the bounding box predictions based on some heuristic confidence level, i.e., we scale the bounding box with 1 -confidence (so we scale the boxes where we are less confident by a larger amount). While this baseline demonstrates a higher completion rate, both the collision rate and the misdetection rate increase significantly, leading to unsafe situations. Further, while our method provides a statistical safety guarantee, the baseline method does not admit any formal assurance.We represent the robot's state as s t = [x, y, v x , v y ] T where x and y are its position in the environment and v x and v y are the respective velocities. For each trial, the robot is initialized around position [4, 0]m (with the origin set to bottom left corner of the room) and has a time horizon of 60 seconds to reach the goal within a 1m radius. The robot replans every 1s in a receding horizon manner using the safe planner described in Section 5. The goals are varied every 10 environments and include positions [2, 7] m, [7, 0]m, and [6, 7] m.Hardware. We use the Unitree Go1 quadruped robot with fully onboard sensing and computation. The robot is equipped with a ZED 2i RGB-D camera and a ZED Box computer attached to the base of the robot as shown in the top row of Figure 7 . The Zed 2i provides the Go1 with point cloud observations with a 70 • field of view and a visibility range of [1, 5] m. The Zed 2i also uses visioninertial odometry to provide accurate positional state estimates in the environment. The Zed Box includes an 8-core ARM processor and a 16GB Orin NX GPU. This allows us to process the point cloud observations in order to produce bounding boxes using the pre-trained 3DETR model [20] . The bounding boxes are aggregated over time to update the estimated free, occupied, and unknown spaces as described in Section 5. The safe planner described in Section 5 is used to output Cartesian velocity commands bounded at a speed of 0.8m/s; these commands are sent from the Zed Box over UDP to the Go1's processor. Our method is implemented real-time on the Zed Box hardware with replanning every 0.5 seconds of which the non-deterministic filter takes 0.00025 seconds to run. The dynamics of the Go1 are estimated using MATLAB's System Identification Toolbox [56] and are provided in Appendix G.2.Environments. We test the robot in 30 different environments, consisting of various chair configurations and geometries in an 8 m ×8 m room. Configurations range from random, occluded goal, occluded chairs, clustered chairs, and narrow paths (approximately 1.8m in width leaving 0.4m of available freespace for PWC to find). Each environment has between 4 and 8 chairs present. See Appendix G.3 and G.4 for the unseen chairs used in testing and the environment configurations respectively. We use a Vicon motion capture system to log the ground-truth placement and bounding boxes of the chairs for each environment.To perform system identification of the Unitree Go1 quadruped robot, we collected trajectories using a Vicon motion capture system. We then used MATLAB's system identification toolbox [56] . Specifically, we provided an initial linear ODE grey box model guess and then used prediction error minimization (PEM) for refinement. The resulting system is shown in (11) where x and y describe the positional state of the robot in the environment, v x and v y describe the respective velocities, and u x and u y describe the respective commanded velocities. As described in Section 7, the robot was tested in 30 unique environments with varying furniture configurations and goals. The following 30 figures show an image of each configuration, accompanied by a bird's-eye map of the obstacle and goal locations. It would be infeasible to consider all possible states in an environment. In practice, we use a samplingbased motion planner and consider a fixed set of samples for our calibration that could be used by any planner.ε is the calibration threshold such that the dataset conditional guarantee (3) achieves the desired (1ϵ)-coverage with probability 1 -δ = 0.99 over the sampling of the calibration dataset.(4) Environment 4 (5) Environment 5 (6) Environment 6 (7) Environment 7 (8) Environment 8 (9) Environment 9(28) Environment 28 (29) Environment 29 (30) Environment 30 


Decision: The paper is an interesting application of CP to object detection. The reviewers bring up several issues that should be addressed in the rebuttal:

- It is unclear which states can be used by the motion planner, is it the same set of states used for calibration?
- The explanation of the non-deterministic filter is unclear.
- The paper should be more self-contained (hard to find Equation 3 in [2])
- The paper would benefit from comparisons to other baselines and ablations
- There needs to be discussion of sim2real for this kind of approach
- Need to discuss the speed of the method

Please see reviewer comments for more details.

-----Post-rebuttal comments----

The authors' rebuttal clarified several issues raised by the reviewers, but the reviewers' ratings remain unchanged. Overall, the need for safety guarantees on learned systems is quite critical, so this paper is in an area that should be given more attention. While there are some limitations to the method (e.g. the fixed set of samples), there is still a contribution here.

Reviews: review: title: title: The paper addresses an important topic of safety of perception-based systems. While interesting, some technical concerns need to be addressed.
paper_summary: paper_summary: The paper proposes the use of Conformal Prediction (CP) to calibrate the output of object detectors. The paper uses standard CP argument to show statistical guarantees of the classification model. The paper demonstrates the efficiency of the proposed method using multiple experiments.
main_review: main_review: - Clarity: The paper is well-written and easy to follow.

- Originality: The use of Conformal Prediction (CP) has recently gained much interest. The paper follows a simple CP argument to provide some notion of the correctness of pre-trained models. Nevertheless, the use of CP in this paper needs to be more carefully elaborated to avoid the common pitfalls in using CP (e.g., hidden assumptions on the model and the distribution).

- Significance: Appropriate. 

- Strengths: 
The paper provides a comprehensive evaluation of the benefits of using the proposed method. 

- Weaknesses:

- - (1) The definition of "environment" is very simplistic. In this paper, the environment affects the obstacles' location and geometry. What about the remaining factors like lighting conditions, weather, ... etc. In practice, one can not get an i.i.d sample that efficiently covers all these environmental changes, which strictly limits the applicability of the proposed methodology.

- - (2) One of the major weaknesses of vision-based models is that ``they can not always detect objects from all angles and under all environmental conditions.'' In other words, vision-based models frequently may not return any bounding box, although the obstacle is in the scene (e.g., what happened in the infamous Uber accident in Arizona). It is not clear how Proposition 1 captures such a case. The current proposition (as it is currently presented) seems to be implicitly assuming that the object detector can always detect the objects but only the bounding box is what needs to be calibrated, which is a very strong assumption given the current state of vision models.

- - (3) The footnote 1 (on Page 3) is vague. Do you mean that one has to fix a set of states to be used during the calibration and use the same set of states to be used by the online motion planner? If this is the case, such a (strong) assumption needs to be explicitly stated in Proposition 2, as motion planners who do not use this set of states during online planning are no longer safe.
questions: questions: The authors should address the weaknesses number (1), (2), and (3) in the review above.
limitations: limitations: 2
review_summary: review_summary: While the paper is interesting, the technical argument behind Proposition 1 and 2 is not thorough and several hidden (and strong) assumptions are not thought out thoroughly.
title: title: A promising uncertainty calibration framework for safe navigation with learning-based perception
paper_summary: paper_summary: To handle the reliability issue in using pre-trained learning-based perception models for planning in navigation, the authors proposed a simple yet effective uncertainty quantication framework based on conformal prediction. The proposed method can provide formal assurance of the correctness in its outputs and is robust against the state distribution shift during closed-loop deployment. Emprical evaluation of the proposed idea has been conducted in both simulation and real-world, showing strong benefits in decreasing collision and mis-detection rates with comparable success rates.
main_review: main_review: ### Strengths 
- The proposed idea is neat and promising in facilitating safe planning while exploiting the pre-trained learning-based perception models. 
- The target problem is highly relevant to the robotic perception, planning, and control communities, facilitating the co-development of an integrated closed-loop autonomous system. 
- The paper is overall good to follow with a clear problem formulation and experiments demonstration except for the description of the method part (detailed below).
- Technically, the idea of ensuring statistical guarantee for outputs of a learning-based model via conformal prediction is not new. Though, exploiting such formal assurance for safe planning highlights the originality of this work. The derived assurance for a safe policy induced by the calibrated perception model in proposition 2 corroborates this aspect. 
- The benefits of the proposed idea are verified in both simulation and the real world in terms of comprehensive metrics such as collision rate, mis-detection rate, success rate, etc.  The results of the proportional relation between the specified confidence thresholds and the actual mis-detection rate further confirm the theoretical statements in the main paper.

### Weaknesses
- For the presentation of the method part:
   1. Lack of brief preliminaries on the core tool used by this work, i.e., conformal prediction in the **main** text, which confuses the reader when evaluating the novelty of the proposed method;
   2. In Section 3.1, it is confusing to highlight only the "closed-loop distribution shift" in bold but not the "generalization." Moreover, it would be preferred if there is a clear definition of each term to replace the a bit wordy description with a large paragraph for conciseness; 
   3. It is a bit awkward for the readers to understand Equation 3. without a proper introduction (maybe this part can be added along with the background section);
   4. Regarding the explanation of the non-deterministic filter, a formal description would be highly recommended for readability. The current explanation is done through a long paragraph and the low-resolution Figure 2., which might be time-consuming for the reader to understand the core idea easily;
- For the technical part, it is hard to find Equation 3 in [2], as mentioned by the authors in the appendix, which should be the key difference to the version of conformal prediction used mostly from my understanding. A more detailed and self-contained description of this part is better to have for convincing the readers who are interested in getting familiar with the technical details;
- Moreover, it seems less convincing not to discuss the covariate shift and general distribution shift for conformal prediction [1]. From my understanding, the so-called "closed-loop distribution shift" is one instance of the covariate shift [1]. I see little value in creating a new term, as it can easily distract readers with such a background. In this sense, if the weighted conformal prediction [1] is one of the principled approaches to address this problem, it would be better to discuss or even include it as a baseline in the experiments;
- On the other hand, empirically, comparing the uncertainty-aware or OOD-aware perception models such as [3] would be sensible to discuss or be added as a baseline in the experiments. Even though this type of method provides only heuristic-based uncertainty quantification, in practice, the implementation is quite similar, i.e., thresholding. The major limitation is the lack of formal assurance, which is addressed by the proposed approach. Comparing them can also hint at whether formal assurance indeed works better for perception-based closed-loop planning. Considering this, adding this part would be, in general, of great interest to the community of uncertainty quantification in robotics.
- Lack of an ablation study of the number of samples for approximating "all" states during calibration: In the footnote on page 3, the "all" possible states are approximated with a fixed set of samples in the planner. 
- Lack of discussion of the sim-to-real gap in using the perception module calibrated in simulation to the real world directly; 
- Last but not least, in the experiments, it's pretty nice to see the empirical validation of the derived proposition 1 in Figure 5. Would it be more appealing and convincing if a similar trend can be observed for the assured safety cost stated in proposition 2;


[1] Barber, Rina Foygel, et al. "Conformal prediction beyond exchangeability." The Annals of Statistics 51.2 (2023): 816-845.

[2] Vovk, Vladimir. "Conditional validity of inductive conformal predictors." Asian conference on machine learning. PMLR, 2012.

[3] Feng, Jianxiang, et al. "Topology-Matching Normalizing Flows for Out-of-Distribution Detection in Robot Learning." 7th Annual Conference on Robot Learning.
questions: questions: My suggestion is to address the points articulated in the list of weakness above.
limitations: limitations: 3
review_summary: review_summary: Due to the insufficient clarity in the presentation and lack of dicussion/experiments for the related work, I would vote for a weak reject at the current stage.
title: title: Review of "Perceive With Confidence: Statistical Safety Assurances for Navigation with Learning-Based Perception"
paper_summary: paper_summary: This paper presents a novel framework, Perceive With Confidence (PWC), that provides statistical safety assurances for navigation in robots using learning-based perception models. By leveraging conformal prediction for uncertainty quantification, the authors ensure robustness to distribution shifts in environments, leading to improved safety in both simulated and real-world scenarios.
main_review: main_review: The paper is well-structured and provides a thorough investigation into the problem of ensuring safety in navigation with learning-based perception. The methodology is sound and well-explained, making a significant contribution to the field of robotics. The writing is clear and concise, making complex concepts accessible. The approach of using conformal prediction for calibrating perception systems to handle distribution shifts is novel and addresses a crucial gap in the current literature. This work is the first to provide such end-to-end safety assurances while being robust to closed-loop distribution shifts. The proposed framework has significant implications for the deployment of robots in real-world environments.

Cons:

 • The framework assumes static obstacles in the environment, which limits its applicability to more dynamic, real-world scenarios where obstacles may move unpredictably. This assumption might hinder the deployment of the system in environments such as urban settings or areas with moving people and vehicles.

 • The integration of the non-deterministic filter and safe planner adds computational overhead, which might affect real-time performance. The paper does not provide detailed benchmarks on the real-time efficiency of the system, particularly under high-speed navigation scenarios or in environments requiring quick decision-making.
questions: questions: • The use of bounding box inflation to ensure safety can introduce conservatism, potentially affecting the robot’s efficiency. Can you discuss potential strategies to mitigate this conservatism without compromising safety?

 • The results indicate that varying the confidence threshold (ϵ) affects the misdetection rate. Can you provide more insights into how different safety thresholds impact overall system performance, and what guidelines would you suggest for selecting an appropriate threshold?
limitations: limitations: 2
review_summary: review_summary: The proposed framework, Perceive With Confidence (PWC), introduces a novel approach to uncertainty quantification through conformal prediction, providing robustness to distribution shifts. The thorough validation through both simulation and hardware experiments demonstrates the practicality and effectiveness of the approach. Despite some limitations, such as the assumption of static obstacles and the conservatism introduced by bounding box inflation, the paper offers substantial advancements in the field of robotics and presents a solid foundation for future research.