Risk-Driven Design of Perception Systems
Authors: Anthony Corso
Sydney Michelle Katz
Craig A Innes
Xin Du
Subramanian Ramamoorthy
Mykel Kochenderfer
Venue: NeurIPS 2022 Accept (2022)
Keywords: Safety-critical autonomy
risk-sensitivity
perception
aircraft collision avoidance
object detection

Abstract: Modern autonomous systems rely on perception modules to process complex sensor measurements into state estimates. These estimates are then passed to a controller, which uses them to make safety-critical decisions. It is therefore important that we design perception systems to minimize errors that reduce the overall safety of the system. We develop a risk-driven approach to designing perception systems that accounts for the effect of perceptual errors on the performance of the fully-integrated, closed-loop system. We formulate a risk function to quantify the effect of a given perceptual error on overall safety, and show how we can use it to design safer perception systems by including a risk-dependent term in the loss function and generating training data in risk-sensitive regions. We evaluate our techniques on a realistic vision-based aircraft detect and avoid application and show that risk-driven design reduces collision risk by 37% over a baseline system.

Risk-Driven Design of Perception Systems
Anthony L. Corso
Sydney M. Katz Department Aeronautics and Astronautics Stanford University Stanford, CA smkatz@stanfordd edu
Department of Aeronautics and Astronautics
Stanford University Stanford, CA acorso@stanfore edu
Craig Innes School of Informatics University of Edinburgh Edinburgh, UK craig.innesded ac.uk
Xin Du School of Informatics University of Edinburgh Edinburgh, UK x.duded.as ac. uk
Subramanian Ramamoorthy
School of Informatics University of Edinburgh Edinburgh, UK
s. rraaamorrth@@ed.coomo
Mykel J. Kochenderfer Department of Aeronautics and Astronautics Stanford University Stanford, CA mykel@staaforee edu
Abstract
Modern autonomous systems rely on perception modules to process complex sensor measurements into state estimates. These estimates then passed controller, which uses them to make safety-critiall decisions. therefore important that we design perception systems minimize errors that reduce the overall safety of the system. We develop a risk-driven approach to designing perception systems that accounts for the effect of perceptual errors on the performance of the fully- integrated, closed-loop system. We formulate risk function to quantify the effect of given perceptual error on overall safety, and show how we can use design safer perception systems by including risk-dependent term in the loss function and generating training data in siskiiensi regions. We evaluate our techniques on a realistic vision- based aircraft detect and avoid application and show that risk-driven design reduces collision risk by 37 over a baseline system.
Introduction
The design of reliable perception systems is a key challenge in the development of safety-critical au- tonomous systems [1], [2]. Modern perception systems are often required to predict state information from complex, high-dimensional inputs such as images or LiDAR data [3]-[5]. This information then passed to a controller, which uses the state estimate make safety-critich decisions. For example, vision-based perception systems have been proposed for detect and avoid applications in aviation [5]. important that perception systems produce accurate estimates, and these systems are typically trained to minimize overall perceptual error using a regression loss set of training data. This training method, however, fails to account for the effect of perceptual errors on the performance of the integrede,, closed-loop control system. In particular, not all errors will have equal
"qqual contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

effect on overall performance. While some errors pose minimal risk, others may lead to catastrophic outcomes [6], [7].
Contribution In this work, we develop a technique to design perception systems that are sensitive to the overall risk of the closed-loop system in which they operate. We formulate risk function that quantifies the downstream effect on safety of making a given perceptual error in given state. We then show how the risk function can be used to design safer perception systems by incorporating into the loss function during training and using it to focus data collection efforts on the most risk-sensitive regions of the state space. Finally, we analyze the impact of our risk- design approach on the safety of a realistic vision-based aircraft detect and avoid system. We show that our approach is able to reduce collision risk by up to 37 % over baseline perception system.
2
Background
This work uses notion of conditional value at risk (CVaR) and a Markov decision process (MDP) formulation to estimate the effect of perceptual errors on the safety of closed-loop system. This section details the necessary background on these topics.
Conditional Value at Risk (CVaR) Let X be bounded random variable and F(x) P(X 2) be its cumulative distribution function. The value at risk (VaR) is the highest value that X is guaranteed not exceed with probability a written as
VaR, (X) min(= x F(x) 2 a}
Using the VaR, we can derive the CVaR for a given a E (0, 1), which represents the expected value of the top - a quantile of the probability distribution over X. More formally,
CVaR, = E|X X > (X)]
If X is random variable that represents cost, CVaR, (X) will correspond to the expected value of the worst a-fraction costs. As a approaches 1, CVaR,, (X) approaches the cost of the worst-case outcome. As approaches O, (X) approaches the expected value of X. CVaR has some desirable mathematical properties [8] that have often made it the preferred risk metric in risk-sensitive domains such as finance [9]-[11] 1] and robotics [8].
Markoy Decision Process (MDP) An MDP is a way of encoding a sequential decision making problem in which an agent's 's action at each time step depends only on its current state [12]. An MDP is defined by the tuple (S, T, C, 7). where is the state space, A is the action space, T(s' | s a) the probability of transitioning to state s' given that are in state and take action a, C(s,, a) is the cost of taking action a in state s, and ? is the discount factor. A stochastic policy for an MDP, written (a | s). denotes the likelihood of taking action in state and induces a distribution over future costs. We define a cost distribution function Z" a) that maps a given state s and action a to random variable representing the of discounted future costs obtained by taking action from state s and subsequently following policy x as follows
(st, at) = C(s),a))+ +
Σ C'C(se, ar))
(3)
emtel
where 8141 T(- sp, ax) and ar ~ T(- sy). To evaluate given policy म. we calculate a state-action value function, denoted Q" (s, .), as the expected sum of discounted future costs
(s E (s)
(4)
In traditional reinforcement learning, Q is the optimization objective [12]; however, in safety-critical domains we may wish evaluate policy based on an upper quantile of the worst-case outcomes. In this case, rather than caleulating the expectation of future costs, we define a new state-action value function Q (s, a) that represents the CVaR of the distribution of future costs as
(5) Intuitively, (s, a) is the expected value of the a-percentile of worst-case outcomes. We note that when a = 0, eq. (5) reduces to eq. (4). One way to solve for (s, a) to first solve for the cost distribution function zd (s, (s,a), which can be done using techniques from distributional reinforcement learning [13]. The CVaR can then be computed using this distribution. Other techniques that do not require explicitly solving for the distribution over costs may also be applied [10], [14].
QT a) = CVaR, (2" a)]

---

System
T
Controller
control
Environment T
Controller
control
Environment
u = g(
1 g(3)
T
Perception System [(o)
Error Model Ne(:
observation o
state
(a) System Architecture
(b) Abstracted Perception MDP
Figure 1: Problem formulation.
Environment
Perception System (f)
Controller (r)
Figure 2: Overview of the inverted pendulum example system.
Approach
consider the setting shown fig. la, in which a system is composed of perception module f and controller g. Given observation E O of the true state S, the perception system produces an estimate the state 3 = (o). The controller produces a control input и = g() and the system transitions to state s' with probability T(s' s, u). We assume that the controller y is given, and we wish to design perception module f such that the overall system satisfies a safety property.
Pendulum example We will use vision-based inverted pendulum problem as running example describe our approach (see Figure 2). We assume that a camera produces noisy image observations of the pendulum's current state [0, w] where 0 is its angle from the vertical and w is its angular velocity. To estimate both position and velocity information from the images, we use two consecutive image frames for each observation. The perception system is a multi-layer perceptron (MLP) that takes in the image observations and produces an estimate of the state. Using the state estimate, the controller selects a torque to keep the pendulum upright. We specify safety property that requires that [0] < 7/4 at all time steps. We use a rule-based controller that satisfies the safety property under perfect perception; however, image noise, image downsampling, and the finite capacity of the MLP make perception errors inevitable, and the goal is to design a perception system that limits the occurrence of high risk emrrr. Appendix A.1 summarizes the inverted pendulum model.
3.1 Overview
The key insight in our approach is abstract the perception system using notional model of perception errors and analyze the effect of specific perception errors on the performance of the closed-loop system. As shown in fig. 1b, we model the errors e & of the abstracted perception system as stochastic policy Te(c s) in an MDP. We note that this representation implicitly takes into account the possible observations for state by directly outputting the errors the perception system could make when processing each observation. The cost function of the MDP is chosen to reffect the risk of being in this way, we can use the policy evaluation techniques outlined in section 2 to evaluate the risk of making perception error e state s. Given that n. is an abstraction of the perception system, will not exactly reflect the true distribution of perception enrors. To account for this mismatch,
state with respect to a safety property. By formulating the problem

---

conservative in our risk estimate by evaluating the policy according to eq. (5). Once computed, the risk function represents the downstream impact of making a given perception error in the current state. This function can therefore be used in supervised learning setting to encourage model to avoid making high-risk errors. We also show that the risk function can be used to identify error-sensitive regions of the state space, from which additional training data can be collected.
3.2
Risk Function
Our goal is to define a risk function p(s, e) that quantifies the risk of making perception error in state when using a controller We model the perception error as a stochastic policy (e s) that reflects a notional distribution over errors in state s. The controller produces an action based on the perceived state s. which is computed from s and e. For additive noise, the resulting transition function for the abstracted pereeption MDP
(s 8, T(s') 8 18.9(s (c))
(6)
The error model Te can be designed to match the performance of a previously trained baseline perception system or using domain knowledge. If we are confident that our choice closely matches the errors we will see when designing the perception system, we can set our risk function to match the state action-value function in eq. (4). However, if we have not yet designed the perception system, there will be a mismatch between our estimate of ne and the true distribution of perception errors.
In these scenarios, we may want a more conservative risk function in which we consider the CVaR of the distribution over future costs when making perception error in state 8 given that all future perception errors are distributed according to Te. Therefore, we select our risk function as follows
Pa (8, c) = Q (s,
where Qu (s, E) refers to the CVaR state-action value function defined in eq. (5) for the abstracted perception MDP. The parameter a smoothly controls the degree of conservatism. When a = 0, the risk function is equivalent to the expected cost when following the notional perception model. When a = 1, we consider only the worst-case sequence of possible perception errors.
To solve for the CVaR state-action value function we apply distributional dynamic programming to compute the distribution over costs. Distributional dynamic programming (alg. 5.3 in Bellemare et al. [13]) resembles value iteration where the Bellman operator is replaced with distributional version to update a parameterized distribution. this work, we approximate the distribution over costs using categorical distribution with a fixed number of discrete costs. Additionally, we discretize the states and actions and use a linear weighting function for local approximation. To extend our method to problems with larger state and action spaces, we could apply function approximation techniques developed in prior work [15]-[17], but we leave this as future work. Once we solve for the cost distributions, can calculate p (s, for any value of a using eq. (2) with minimal additional computation.
Pendulum example For the inverted pendulum system, we model the perception errors = [ea. ew] as additive noise on the true state (& + <) distributed according to a multivariate Gaussian distribution with zero mean and diagonal covariance. The cost function penalizes the absolute deviation from the vertical [@). See appendix A.2 for more detail. Figure shows the values of the pendulum risk function at the state higher value of a will put more weight on worst-case perception errors in the future, so as a increases, the overall risk values increase. The risk function also provides a notion of which perception errors are most risky in a given state. For example, the optimal action when s = [0.2, 0.0] is to apply a negative torque to move the pendulum closer to upright, but if perception errors are negative, the resulting& may indicate that the pendulum is at a smaller angle and already moving in the negative direction as shown in the leftmost image fig. 3. This estimate may cause the controller to produce a smaller torque or even a torque in the positive direction, leading to high risk of falling over.
[0.2, 0.0] over the set of errors for multiple values of A
3.3 Risk-Sensitive Loss Function
Perceptual errors are inevitable whenever there is sensor noise, occlusions, limited training data, or limited model capacity. Our goal is to ensure that the errors that are made by the perception sy system are in the direction of lower, rather than higher, risk. We incorporate this notion into our design by

---

po c.
66 (00.2.000)
P63 (10.2
0.6
0.5
0.4
0.5
0.2
0.4-0.2
0
0.2 0.4 0.4-0.2
0.2 0.4 0.4-0.44
0.4-0.2 0
0.2
0.4
c0
60
Figure 3:
Values of pT 0.0. e) for different values a. The state and an example error are depicted on the left.
formulating a risk-sensitive loss function. Let [(s, 8) be the standard loss function for the baseline perception system such mean squared error for regression tasks or cross entropy for classification tasks. We define a risk-sensitive loss function for a single datapoint with additive noise as follows
(s) = [(s, + Ap= (s, - s)
(8)
where À is hyperparameter that controls the relative weighting of each term. The first term in the loss function pushes all perception errors toward zero, while the second term seeks to minimize the risk of the current errors that the model makes. We note that this is different from simply minimizing perception errors in states with high risk values. If the risk function outputs the same value for a given state regardless of perception error, the gradient of the risk term in the loss function will be zero
Pendulum example For the inverted pendulum problem, we use mean squared error baseline loss function, which penalizes positive and negative errors equally. However, positive and negative of the same magnitude do not necessarily have an equal effect on the overall risk the closed-loop system. For example, consider the results shown in fig. 3 where the pendulum is positioned at
angle of 0.2 from the vertical. The risk function shows that a perception error co 0.2 (0 0.4) is less risky than a perception error of eg = -0.2 ( 0.0). Intuitively, predicting that the pendulum tipped further in the positive direction will still result in a negative torque move the pendulum closer to upright. In contrast, predicting that the pendulum perfectly upright will cause the controller to output zero torque. The second term in eq. (8) accounts for this effect.
3.4 Risk Data Generation
In addition considering the directionality of the errors, can use the risk function we formulated in section 3.2 to identify states that are especially sensitive to perception errors and generate more training data in those regions. In effect, this process gives higher weight risky states during training. Since training with regression loss will push all errors toward zero, we define risky states as states which making a nonzero perception error will result in high risk value compared to the risk of zero error. We use weighting function Wa (s) to capture this notion as follows
Wa (s) max ae (s,e) e) pa (s, 0)
cEE
We note that w. (s) is not the same as (and not necessarily correlated with) the risk of being state Instead, (s) represents the risk of making nonzero perception error in state For instance, states in which failure is inevitable regardless of perception error will have high risk values but low rejection sampling with (s(s)
weight assigned to them according to w. (s). To sample data, as the likelihood function.
Pendulum example Figure 4 illustrates each term in eq. (9) for the inverted pendulum example problem. The lower left and upper right regions of the state space represent states in which the pendulum is at high angle and moving in a direction away from the vertical. Due to underactuation of the pendulum system, the states these extremes are guaranteed to lead to failure regardless of perception error. States in the central band of each plot, on the other hand, constitute states in which the pendulum likely remain upright as long as proper control applied. This band widens when we consider the effect of having zero perception error compared to the worst-case perception
5

---

max. EF Po (s,c)
Pg (s
(s)
2
0.8
1
0.6
6o (radis)
0
0.4
-1
0.2
0.5
0
0.5
0.5
0.5
-0.5
0
0.5
0 (rad)
(rad)
0 (rad)
Figure 4: Illustration of eq. (9).
error. By subtracting these quantities, we can understand in which states having zero perception emror most critical. Given a limited data budget, we can focus our data collection on these states by sampling data according to W. (s).
3.5 Hypotheses
We design the experiments in this work to test the following two hypotheses:
H1: A perception module trained using the risk-sensitive loss function proposed in sec- tion 3.3 will result in safer closed-loop system than one trained using the baseline loss the same dataset.
H2: A perception system trained on data sampled according the risk-driven data generation technique proposed in section 3.4 will result in safer closed-loop system than one trained on data uniformly sampled throughout the state space.
Pendulum example We evaluate the inverted pendulum by measuring the mean time to failure 100 trajectories each with 500 time steps from random initial states. We report the mean and standard етrог of this metric over trials with 500 being the maximum attainable value. To test H1, we train baseline perception system using a mean squared error loss function and risk-driven perception system using the loss function described in section 3.3 on 10,000 uniformly sampled data points. make the perception problem more challenging, we add noise to the images. The risk sensitive loss function improved the mean time to failure from the baseline of 234 + 62 to 492 + 6. To test H2 on the inverted pendulum system, we limit the perception system training examples. We train baseline perception system on data sampled uniformly throughout the state space and a risk-driven perception system on data sampled according to the weighting function shown in fig. 4. The mean time to failure for the baseline system is 152 + 89, while the time to failure the system trained the risk-driven data was 500 for all trials. See appendix A.3 for full results.
data budget of 50
Vision-Based Detect and Avoid Application
Aircraft collision avoidance systems use information gathered from sensors to detect intruding aircraft and issue advisories for safe collision avoidance maneuvers [18]. Traditional sensors used for surveillance and tracking include ADS-B B, onboard radar, and transponders [19]; however, autonomous aircraft require additional sensors both for redundancy and to replace the visual acquisition typically performed by the pilot. For this reason, vision- based traffic detection systems have been proposed, in which intruding aircraft are detected from images taken by camera sensor mounted on the aircraft [5], [20]. In this section, will apply risk-driven design techniques improve the safety a vision-based detect and avoid (DAA) system. The code for this work can be found at wtttt//w...ooosssssssssieiiieeee
4.1 Problem Setup
Figure outlines the components vision-based DAA system. camera mounted on the aircraft (which we refer to the ownship) produces image observations of its surroundings. These images

---

Environment
Controller g
Perception System
intrnder
ownship
Figure Overview of the vision-based detect and avoid system application.
are then passed through a perception system to produce an estimate of the state for the controller, The collision avoidance controller g. which loosely based on the ACAS X family of collision avoidance systems [18], commands vertical maneuvers based on an estimate of the relative position and velocity of intruder aircraft. Specifically, the state is defined as [p. h h, aprev, T] where p is binary the relative altitude of the intruder, h the
variable describing whether an intruder is present, relative vertical rate of the intruder, aprev is the previous action, and т is the time to loss of horizontal separation. The controller selects an action clear of conflict. If p O, the action is always COC.
{coc, CLIMB, DESCEND) where COC represents
We assume that the state estimate for the controller is produced using two steps. First, the image is passed through perception network that responsible for detecting other aircraft (which we refer intruders) and producing bounding box. Once intruder has been detected, the ownship interrogates it to determine the rest of the state. We focus our risk-driven design efforts on the detection component of the perception system. We train baseline network using the YOLOv5 algorithm [21]. [22] on 10,000 simulated images in which the intruder location sampled uniformly within the ownship field of view. For additional details on the controller, network architecture, computational resources, and training, see appendix B.
4.2 Risk-Driven Design
An object detector can make two types of errors: false positives and false negatives. While false positives have an effect on the efficiency of the system, they do not affect the risk of a collision. We therefore focus on quantifying the risk of of false negatives in the presence of intruder. We define the perception error as zero when an intruder is properly detected and one when is missed, leading perceived state
[1 -,,,,, h, h, Oprev 7]
(10)
The notional error model (E 8) is Bernoulli distribution with success probability based on the detection performance of the baseline model (see appendix for more detail). To solve for Q (8,e), discretize the state space and apply approximate dynamic programming.
Risk-sensitive loss function box coordinates, the detection network outputs an resenting the model's confidence that an aircraft The model detects an aircraft after checking if á exceeds threshold; however, this check poses numerical chal- lenge when differentiating the risk sensitive loss since the
In addition
outputting the bounding objectness score i rep- present in the frame.
200
gradient is undefined at the decision threshold and zero every where else. To avoid this problem evaluate the risk و directly by interpolating between the risk of not detecting and detecting the intruder
& (z))
-200
P (s = (T) (s, 0) + (1 -) (s,
(11)
The gradient risk function in eq. (11) with respect to is equal to the risk weight, so the model will be incentivized to be more confident in states where a detection has significant effect overall risk.
10
20
30
40
7
(S)
Figure 6: DAA weighting function.
7

---

Bascline
Risk Loss
Risk Data
Both
600
No. of NMACs
0.8
400
0.6
0.4
200
0.2
20
40
60
80
100
120
140
Risk
Figure 7: Number of NMACs and cumulative risk for each perception system.
Risk-driven data generation Figure 6 shows the weighting function when 0 and an intruder is present. States with high weights correspond to states where not detecting the intruder is significantly more risky than detecting the intruder. While collision is imminent in states with low values of and |//. these states have low weights since a collision will occur regardless of whether the intruder is detected. The states with the highest risk weights are states in which there just enough time to resolve an impending collision as long as the intruder is detected.
4.3 Evaluation
The safety of aircraft collision avoidance systems is often assessed using Monte Carlo analysis on airspace encounter models [23], [24]. Encounter models are probabilistic representations typical aircraft behavior during close encounter with another aircraft. To analyze the safety of particular collision avoidance system, we can simulate the system on a set of encounters and count the number of times near mid-air collision (NMAC) occurs. We define NMAC as simultaneous loss of aircraft separation to within 50 m vertically and 100 m horizontally. For the evaluation in this work, model consisting of 1000 pairwise encounters in which the ownship and intruder follow straight line trajectories with various relative geometries. All encounters result in an NMAC if no collision avoidance actions are taken (see appendix B.5).
We trained perception networks using four perception design techniques: no risk awareness (baseline), risk-sensitive loss function, risk-driven data generation, and both risk-sensitive loss function and risk-driven data generation. We performed three trials for each perception design method. The first section of fig. shows the number of encounters that result in NMAC for each technique. In support of H1, training with the risk-sensitive loss function (blue) results in a slight increase in safety. In support of H2, training on data generated using the weighting function (red) results in 33 % decrease in the number of NMACs over the baseline. With a 37 % decrease in the number of NMACs compared to the baseline, the perception systems that were developed using both risk-driven techniques (purple) are slightly safer than using either technique individually. Figure also shows the cumulative distribution
the risks of the errors made by each perception system the encounter set. This result implies that even small shifts in overall risk can have significant impacts on safety.
Figure 8 shows example encounter that resulted in NMAC for the baseline system but was resolved by the risk-sensitive systems. The risk-sensitive system (purple) detects the intruder aircraft sooner and more often than the baseline. Moreover, the plot at the bottom of the figure shows that the risks of the perceptual errors made by the baseline system are much higher than the risks the errors made by the risk-sensitive system.
5 Related Work
We discuss prior work in perception system evaluation, risk-aware control, and task-aware prediction.
Perception system evaluation The nigh-dimensional, complex nature of perception inputs makes closed-loop safety evaluation of perception systems especially challenging. This challenge was first noted in early work on computer vision, in which systems were evaluated by propagating uncertainty in set of input images through each computer vision component [25], [26]. Similarly, a number of recent works have focused on finding adversarial perturbations that will cause misclassification

---

4,000
intruder
Baseline
3,500
3,000
Risk Sensitive
2,500
North (m)
2,000
1,500
Risk Altitude (m)
1,340
1,000
1,300
500
100
ownship
80
0
500
1,000
5 10
15
20
25
30
35
40
East (m)
Time (8)
Figure Example encounter that was resolved by the risk-sensitive systems but resulted in an NMAC for the baseline system. The red line denotes the intruder trajectory, while the gray and purple lines denote the ownship trajectory when using the baseline and risk-sensitive perception systems respectively. Left: overhead view of the encounter. Bottom right: altitude of each aircraft over time along with the risk of each perception error. The dotted markings indicate time steps which the intruder was detected. Top right: perception outputs of each system at four snapshots in time.
of single input images [27]. Julian et al. [6], in contrast, use adaptive stress testing (AST) [28] to find sequences of adversarial perturbations that will lead vision-based aircraft taxi system to failure rather than focusing on single images isolation. However, this approach is only able to find failures and cannot guarantee safety if no failures are found. Katz et al. [29] provide approximate formal guarantees of closed-loop safety properties by using generative model to approximate the set of plausible inputs to perception system. Dreossi al. [30] address the complexity of analyzing the closed-loop system by analyzing the controller and perception system separately before combining the results to find failures in the overall system. This work draws inspiration from this decoupled approach.
Risk-aware control In ccritccl domains, it is desirable to design controllers that perform well even worst-case conditions. This problem has been addressed through safety-constrained MDPs [31] and shielding techniques in reinforcement learning [32]. Other risk-aware control approaches use notion CVaR to design controllers. CVaR MDPs, instance, have been used to produce control policies that are robust to modeling errors and worst-case outcomes [10], [14]. Distributional reinforcement learning approaches estimate the distribution of returns [15]. Using the estimate the return distribution, it is possible to compute risk-related metrics such CVaR and produce risk-aware reinforcement learning policies [11]. [16], [17], [33]. In this work, rather than using a notion risk to design control policies, use it to design safe perception systems.
Task-aware prediction Previous work has explored the benefit of using task aware metrics to inform the design of predictive models. For example. Lambert et al. [34] note that more accurate dynamics models reinforcement learning do not necessarily correlate with higher rewards, and Bansal et al. [35] show improved performance when learning dynamics model that achieves the best control rather than learning the most accurate model. Multiple techniques have been proposed to augment traditional training loss functions with task-specific metrics to design models that are not only accurate

---

but also satisfy domain constraints [36]-[38]. In the context of trajectory prediction, McAllister er al. [39] propose control-ap aware prediction objectives, which take into account the downstream effects of the predictions on controller performance. These works motivate the need for task-aware objectives but are not used with image-based perception. In the context of perception, Creiffenhagen el [40] introduce the idea of connecting perception errors to overall system performance for the design and evaluation of a system that uses traditional computer vision [40]-[42]. Furthermore, Philion er al. [43] develop planner-centric metrics for an machine-learnig based object detection system and show their benefit over traditional metrics used to evaluate object detectors. However, they focus only on the evaluation of perception systems, while this work focuses on their design.
6 Conclusion
In this work, we presented methodology for risk-driven design of safety-critical perception systems. We formulated a risk function that measures the effect of perception errors on the closed-loop performance of the fully-integrated perception and control system. We then showed how to use that risk function during the design process by incorporating it into the loss function and developing risk-driven data generation technique. We demonstrated our approach on a realistic vision-bai based aireraft detect and avoid system and showed that our techniques could increase safety by 37 over a baseline system. We note that while the methods presented here focus on designing safe perception systems, they do not represent safety guarantees. Therefore, perception systems designed in this manner should still be put through additional testing and safety validation before deployment. A limitation of this work that we assume that the perception system is Markov, which limits the applicability of our technique to perception systems that use filtering to produce state estimates. Future work will address this limitation.
Acknowledgments and Disclosure of Funding
The NASA University Leadership Initiative (grant #80NSSC20M0163 provided funds to assist the authors with their research. This research was also supported by the National Science Founda- tion Graduate Research Fellowship under Grant No. DGE- Any opinion, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of any NASA entity the National Science Foundation. This work was also supported by a grant from the UKRI Strategic Priorities Fund to the UKRI Research Node on Trustworthy Autonomous Systems Governance and Regulation (EP/V026607/1, 2020-2024).
References
[1] F. Rosique, P. J. Navarro, C. Fernández, and A. Padilla, "A systematic review of perception system and simulators for autonomous vehicles research," Sensors, vol. 19, no. p. 648. 2019.
[2] J. Van Brummelen, M. O'Brien, D. Gruyer, and H. Najjaran, "Autonomous vehicle perception: The technology of today and tomorrow." Transportation Research Part C: Emerging Technologies, vol. 89, pp. 384-38448 406, 2018.
[3] S. Jung, Hwang, H. Shin, and D. H. Shim, "Perception, guidance, and navigation for indoor autonomous drone racing using deep learning." " IEEE Robotics and Automation Letters, vol. 3, no. 3, pp. 2539-2544, 2018.
[4] Y. Li and J. Sranez-Guzman, "Lidar for autonomous driving: automotive lidar and pereeption systems," IEEE Signal Processing Magazine, vol. 37, no. 4, pp. 50-61, 2020.
The principles, challenges, and trends for
[5] R. Opromolla and G. Fasano, "Visual-basee obstacle detection and tracking, and conflict detection for small UAS sense and avoid," Aerospace Science and Technology, vol. 119, p. 107 167, 2021.
[6] K. D. Julian, R. Lee, and M. J. Kochenderfer, "Validation of image-based neural network controllers through adaptive stress testing." in IEEE International Conference on Intelligent Transportation Systems (ITSC), 2020.
[7] A. Corso, Du, K. Driggs-Camphell, and M. Kochenderfer, "Adaptive stress testing with reward augmentation for autonomous vehicle validation," IEEE International Conference on Intelligent Transportation Systems (ITSC), 2019.
[8] A. Majumdar and M. Pavone, "How should a robot assess risk? Towards robotics," in Robotics Research, Springer, 2020, pp. 75-84.
an axiomatic theory of risk in
10

---

[9] G. C. Pflug, "Some remarks on the value-at-risk and the conditional value-at-risk." in Probubilistic constrained optimization, Springer, 2000, pp. 272-281.
[10] Y. Chow, A. Tamar, S. Mannor, and M. Pavone, "Risk-sensitive and robust decision-making: A CVaR optimization approach," Advances Neural Information Processing Systems (Neur[PS), vol. 28, 2015. [11] R. Keramati, C. Dann, A. Tamkin, and E. Brunskill, "Being optimistic be conservative: Quickly learnin a CVaR policy," in AAAI Conference on Artificial Intelligence, vol. 34, 2020, pp. 4436 4443.
[12] M. J. Kochenderfer, Decision Making Under Uncertaint:: Theory and Application. MIT Press, 2015.
[13] M. G. Bellemare, W. Dabney, and M. Rowland, Distributional Reinforcement Learning. MIT Press, 2022, org.
http:// http://www.cce wwtp:.disrrbbutionneee..
[14] W. B. Haskell and R. Jain, "A convex amalytic approach risk-aware Markov decision processes." SIAM
Journal on Control and Optimization, vol. 53, no. 3, pp. 1569-1598, 2015.
[15] M. G. Bellemare, W. Dabney, and R. Munos, "A distributional perspective reinforcement leaming." in International Conference on Machine Learning, 2017. pp. 449-458.
[16] W. Dabney, M. Rowland, M. Bellemare, and R. Munos, "Distributional reinforcement learning with quantile regression." " in AAAI Conference on Artificial Intelligence, vol. 32, 2018.
[17] W. Dabney, G. Ostrovski, D. Silver, and R. Munos, "Implicit quantile networks for distributional reinforcement learning," in International Conference on Machine Learning, 2018, pp. 1096-1105.
[18] J. Kochenderfer, J. E. Holland, and J. Chryssanthacoppolls. "Next-generation airborne collision avoidance system." Massachusetts Institute of Technology- Lincoll Laboratory. Tech. Rep., 2012.
[19] M. P. Owen, A. Panken, R. Moss, Alvarez, and C. Leeper, "ACAS Xu: Integrated collision avoidance and detect and avoid capability for UAS," in Digital Avionics Systems Conference (DASC), IEEE, 2019, pp. 1-10.
[20] J. James, J. J. Ford, and T. L. Molloy, "Learning to detect aireraft for long- range vision-based sense-and and- avoid systems," IEEE Robotics and Automation Letters, vol. 3, no. 4. pp. 4383 4390, 2018.
[21] J. Redmon. S. Divvala, R. Girshick, and A. Farhadi, "You only look once: Unified, real-time object detection," in IEEE Computer Society Conference Computer Vision and Pattern Recognition (CVPR),
Jun. 2016.
[22] G. Jocher, A. Chaurasia, A. Stoken, J. Borovec, NanoCode012, Y. Kwon, TaoXie, Fang, imyhxy. Michael, Lorna, A. V. D. Montes. J. Nadar. Laughing, tkianai, yxNONG. P. Skalski, Wang, A. Hogan, C. Fati, L. Mammana, Alex AlexWang1900 D. Patel, D. Yiwei, F. You, J. Hajek, L. Diaconu, and M. T. Minh, wftralytics/yolov:: v TensorRT, TensorFlow Edge TPU and OpenVINO Export and Inference, version v6.1, 1, Feb, 2022.
[23] B. J. Chludzinski, "Evaluation of TCAS II version 7.1 using the FAA fast-time encounter generator model," Massachusetts Institute of Technology, Lincoln Laboratory, Project Report ATC-346, 2009.
[24] J. F. Holland, M. J. Knchenderfer, and W. A. Olson, "Optimizing the next generation collision avoidance system for safe, suitable, and acceptable operational performance," Air Traffic Control Quarterly, vol. 21,
no. 3. pp. 275-297, 2013.
[25] X. Liu, Kanungo, and R. M. Haralick, "On the use of error propagation for statistical validation of computer vision software," " IEFE transactions on pattern analysis and machine intelligence, vol. 27, no. 10, pp. 1603-1614, 2005.
[26] R. M. Haralick, "Performance characterization in computer vision." in British Machine Vision Conference, Springer. 1992 pp. 1-8.
[27]
G. R. Machado, E. Silva, and R. R. Goldschmidt, "Adversarial machine learning in image classification:
A survey toward the defender's perspective." ACM Computing Surveys (CSUR), vol. 55, no. 1, pp. 1-38, 2021.
[28] A. Corso, R. J. Moss, M. Koren, R. Lee, and J. Kochenderfer, "A survey of algorithms for black- box safety validation of cyber-physical systems." Journal of Artificial Intelligence Research, vol. 72, no. 2005.02979, pp. 377-428, 2021.
[29] S. M. Katz, A. L. Corso, C. A. Strong, and M. J. Kochenderfer, "Verification of image-based neural network controllers using generative models," in Digital Avionics Systems Conference (DASC), 2021.
[30] T. Dreossi, A. Donzé, and S. A. Seshia, "Compositional falsification of syber--hysice systems with machine learning components." Journal of Automated Reasoning, vol. 63, no. 4, pp. 1031-1053, 2019.
[31] A. Wachi, Y. Sui, Y. Yue, and M. Ono, "Safe exploration and optimization of constrained MDPs using Gaussian processes," in AAA/ Conference on Artificial Intelligence, vol. 32, 2018.
[32] M. Alshickh, R. Bloem, R. Ehlers, B. Könighofer, S. Nickum, and U. Topcu, "Safe reinforcement leamning via shiclding," in AAAI Conference on Artificial Intelligence, vol. 32, 2018.
[33] Y. Chow and M. Ghavamzadeh, "Algorithmss for CVaR optimization in MDPs." Advances in Neural Information Processing Systems (NeurIPS), vol. 27, 2014.
[34] N. Lambert, B. Amos, O. Yadan, and R. Calandra, "Objective mismatch in model-hased reinforcement learning." in Learning for Dynamics and Control, PMLR, 2020. pp. 761-770.
11

---

[35] S. Bansal, R. Calandra, T. Xiao, S. Levine, and C. J. Tomlin, "Goal-driven dynamics learning via Bayesian optimization." in Conference Decision and Control (CDC), IEEE, 2017, pp. 5168-5173.
[36] C. Innes and R. Ramamoorthy, "Flaborating learned demonstrations with temporal logic specifications"" in Robotics: Science and Systems, 2020.
[37] M. Fischer, M. Balunovic, D. Drachsler-
I. Gehr, C. Zhang. and M. Vechev, "DL2: Training and querying neural networks with logic." International Conference on Machine Learning, K. Chaudhuri and R. Salakhutdinov, Edss, ser. Proceedings of Machine Learning Research, vol. 97, PMIR, M,, Sep. 2019, pp. 1931-1941.
[38] K. Leung. N. Aréchiga, and M. Pavone, "Backk-poppggae through signal temporal logic specifications: Infusing logical structure into gradient- based methods." in International Workshop on the Algorithmic Foundations of Robotics, Springer, 2020, pp. 432-449.
[39] R. McAllister, B. Wulfe, J. Mercat, L. Ellis, S. Levine, and A. Gaidon, "Control-aware prediction objectives for autonomous driving," in IEEE International Conference Robotics and Automation (ICRA), 2022.
[40] M. Greiffenhagen, D. Comaniciu, H. Niemann, and V. Ramesh, "Design, analysis, and engineering of video monitoring systems: An approach and case study." " Proceedings of the IEEE, vol. 89, no. 10,
pp. 1517, 2001.
[41] M. Greiffenhagen, Ramesh, D. Comaniciu, and H. Niemann, "Statistical modeling and performance characterization of a real-time dual camera surveillance system," in IEEE Computer Society Conference Computer Vision and Pattern Recognition (CVPR). 2000, pp. 335-342.
[42] M. Greiffenhagen, V. Ramesh, and H. Niemann, "The systematic design and analysis cycle of a vision system: A case study in video surveillance," in IEEE Computer Society Conference on Computer Vision and Puttern Recognition, 2001.
[43] J. Philion. A. Kar, and S. Fidler, "Leariin to evaluate perception models using planner-centric metrics,"
in IEEE Computer Sociery Conference Computer Vision and Pattern Recognition (CVPR), 2020,
pp. 055-544 064.
[44] S. Katz, A. Corso, S. Chinchali, A. Elhafsi, A. Sharma, M. Pavone, and Kochenderfer, "NASA ULI aircraft taxi dataset," Stanford Digital Repository, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] see section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We note in section 6 that the approach does not provide any safety guarantees.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main exper- imental results (either in the supplemental material or as URL)? [Yes] All code is attached in the supplementary material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See appendix A and appendix B
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] We include standard error for both experimental domains across multiple trials
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See appendix A and appendix B
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We use and cite YOLOv5 [22]
12

---

(b) Did you mention the license of the assets? [Yes] As we note in appendix B, YOLOv5 under the GNU general public license
(c) Did you include any new assets either the supplemental material or as URL? [Yes]
We include trained aireraft detection networks in the supplementary material.
(d) Did you discuss whether and how consent obtained from people whose data you're using/curating? [N/A] We did not collect any data from people. Only flight simulator data from Xplane 11.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] see previous
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] No human subjects
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "pk5isHsC5gR", "reviewer": "Reviewer_MFas", "summary": "The paper proposes a risk driven loss and augmentation technique to improve safety performance of image based control systems. They introduce an MDP between the perception model and the controller which quantifies the risks associated with action taken at a given state, and the states which are risky. The MDP is used to quantify risk and is learned with distributional reinforcement learning. Evaluation with pendulum and a simple aircraft avoidance system shows improved results compared to vanilla perception models. ", "questions": "I have no questions", "limitations": "Same as weaknesses", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The idea is novel: use of a Q-function to assess the risk of a perception module is a departure from prior risk-sensitive systems. \n- The paper is well written. The problem statement is clear, there is sufficient background provided on the method, and the implementation details make sense. Use of inverted pendulum as an example makes the paper easy to follow.\n- The evaluation with an airplane avoidance system is sufficiently complex to evaluate the hypotheses posed in the paper.\n\nWeaknesses:\n- There is no details provided on how Q-learning is done. Do you use a neural network? What are the hyper-parameters? If not a neural network, what is the computational complexity?\n- The paper uses discrete systems throughout. I suspect this is because of the distributional Q-learning algorithm. The paper should mention this as a limitation and perhaps discuss methods on how it can be addressed.\n- The evaluation examples are simple. Evaluation on prominent benchmarks or simulators in autonomous vehicles or other realistic control systems would make the paper more convincing. \n- Figure 2 is difficult to read, especially the line plots on the right. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "lK_BZGXaVvJ", "reviewer": "Reviewer_nsDT", "summary": "The authors propose a method for quantifying the control risk induced by perception errors (state-estimation errors) in partially-observable systems (e.g. image observations).\nThe method assumes a dynamics model and a controller are already available and focuses on providing risk awareness to the existing system.\nWith a predefined probabilistic model of perception errors, risk is defined as the *Conditional Value at Risk* (CVaR) of the distribution of cumulative rollout cost (i.e. cost-to-go, or return) under the uncertainty of the assumed dynamics, control policy and perception error model.\nThe cost-to-go distribution is estimated using a categorical distribution, discretizing the cost space.\nOnce the cost-to-go density is available, the CVaR risk metric can be evaluated.\nIt is then used as a secondary objective for training a refined version of the perception model / state estimator, as well as for active data acquisition when training that component.\nThe method is evaluated on the image pendulum (simulated) and on an aircraft collision avoidance data set, demonstrating significant improvement in risk-averse behavior.\n", "questions": "See above.", "limitations": "I don't see any major negative societal impact. Some of the method's limitations are acknowledged in the conclusion.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- Accounting for estimation errors is a good idea. Blindly using bad state estimators (what's referred to as a perception system in the paper) for MPC control can have significant negative impact on performance, so the contribution appears pragmatic.\n- The method appears generic and could be applied to other settings.\n- The individual ablations of the active data acquisition and the risk-sensitive loss are very appreciated.\n- The running pendulum example throughout the manuscript is intuitive and provides clarity w.r.t. the different aspects of the method.\n- Quantifying the relative perception risk of states, by subtracting the risk associated with a perception error of 0, seems like a useful idea.\n\n**Weaknesses**\n- The method explicitly targets partially-observable systems, with an observation space (e.g. images) that differs from the Markovian state-space (e.g. joint angles) of the system.\nThe ideal targetted problem should be a *Partially-Observable Markov Decision Process* (POMDP) [1], whereas the authors assume an MDP for control.\nA discussion of how the method deviates from the POMDP paradigm is necessary to make its limitations clear, and is currently missing.\n- For proper POMDP treatment, state estimators generally need to account for the history of observations (e.g. through probabilistic filtering), or alternatively the policy can be recursive and take the history of observations into account [1]. Tying this into my previous remark, I think this needs to be mentioned in the paper; from what I understand the considered visual perception systems map only the current observation to a deterministic state (sec. 3). The authors briefly touch on this aspect in the conclusion, but I think this deserves a more detailed discussion.\n- The objective for the refined state estimator is mixing state estimation and control, because the added risk-averse term still tries to maximize the CVaR of the cumulative cost under the assumed controller.\nHowever, the refined module is still framed as a perception system / a state estimator, which dillutes the concept in my opinion.\nI would suggest positioning the trained model in another way, e.g. treating it as a sub-parameterization of the overall policy which would include the controller $g$ as well.\n- The method assumes that data is available in advance and can be queried freely to optimize the risk objective or perform active data acquisition. In general, this is not the case for on-line RL, where an agent would need to query the environment consecutively and data acquisition can be expensive.\n- I would have preferred more systems explored in the experimental section. The method is generic, and that would support its broad applicability.\n\n**Further remarks**\n- The overview in section 3.1 could point out what will happen with the observations to prepare the reader for what's to come. Currently it only discusses quantifying the state estimation errors, but not how that fits together with the existing state estimator. Diagram in Fig. 1b is also missing the observations alltogether.\n- $Z^{\\pi}$ is described as a \"cost distribution function\" in sec. 2, but if I interpret the equations correctly it denotes the random variable of the infinite-horizon return, for which expectations, CVaR, etc. can be computed. If my interpretation is correct, I think it could be phrased better.\n- I am not convinced framing the perception error model as a policy is the best idea. The original policy is still involved in the rollout in the form of $g$, the controller, and having two policies is confusing. Why not treat it as an additive error model on top of the transition, for example?\n\n\nSince the method appears generic and I like the idea of quantifying the future state estimation errors of the system, I'm leaning towards a positive assessment.\nHowever, there are conceptual issues that need to be addressed in the manuscript and I find the evaluation of the method can be improved by considering further systems.\n\n**References**\n\n[1] Bertsekas, D., 2012. Dynamic programming and optimal control: Volume I (Vol. 1). Athena scientific.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZmZO5rh3XDE", "reviewer": "Reviewer_ndHK", "summary": "The paper articulates a risk-driven approach for designing perception systems that accounts for the effect of perceptual errors on the performance of the fully integrated, closed-loop system. The key contribution is the use of conditional value at risk (CVaR) and a Markov decision process (MDP) formulation to estimate the effect of perceptual errors on the safety of a closed-loop system. A model of errors of an abstracted perception system is used as a stochastic policy of the MDP. Further, the CVaR of the distribution of future costs is used as the state-action value function leading to a policy that is based on the upper quantile of the worst-case outcomes.  The risk function computed represents the downstream impact of making a given perception error in the current state and is used in a supervised learning setting to encourage a model to avoid making high-risk errors. The risk function is also used to identify error-sensitive regions of the state space, from which additional training data can be collected. The concept is illustrated in a toy example involving pendulum dynamics and in a vision based aircraft detect and avoid application and it is shown that the collision risk is reduced by 37% over a chosen baseline.", "questions": "It was not clear from reading the paper if the ACAS example datasets you use are for a simulated setting or real.  How do you see the work translating to real-world settings?   How do you validate your model assumptions ?  For instance, even in this simple detection scenario, the bernoulli random variates used for probability of misdetection and probability of false alarm are complex functions of the trained yolomodel as well as the current sensing conditions (e.g. scene state, sensor noise, etc).   ", "limitations": "It is nice to see that the authors acknowledge that the concepts presented in the paper do not provide safety guarantees.  Can you comment about what specific gaps are there from taking this work to a full blown deployable system?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality/Novelty:  The idea of systems level performance modeling and how perception errors can be linked to control for achieving particular end to end goals has been studied before. (See for instance: Greiffenhagen et al, Statistical Modeling and Performance Characterization of a Real-Time Dual Camera Surveillance System. CVPR 2000: 2335-2342.).  In this sense, the concept in the paper is well known and incremental. Other aspects including formulation of the risk sensitive loss function, dealing with differentiability issues in the loss formulation, data generation, illustration of utility in the application setting are novel.  \n\nReference to Past work:  A number of papers have appeared in the computer vision literature in the 90’s and the 2000’s exploiting uncertainty propagation / distribution propagation for system identification. See haralick.org for several papers on performance characterization in computer vision context. The paper does not refer to any of these past literature.  \n\nSignificance of work:  The work incorporates a risk sensitive loss function in the perception module design and is an illustration, in the modern ML (with supervised learning) context, of how error models in perception should be linked to overall risk in the control scenario.  The example case studies are narrow but sufficiently illustrate the concept.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept