Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper articulates a risk-driven approach for designing perception systems that accounts for the effect of perceptual errors on the performance of the fully integrated, closed-loop system. The key contribution is the use of conditional value at risk (CVaR) and a Markov decision process (MDP) formulation to estimate the effect of perceptual errors on the safety of a closed-loop system. A model of errors of an abstracted perception system is used as a stochastic policy of the MDP. Further, the CVaR of the distribution of future costs is used as the state-action value function leading to a policy that is based on the upper quantile of the worst-case outcomes.  The risk function computed represents the downstream impact of making a given perception error in the current state and is used in a supervised learning setting to encourage a model to avoid making high-risk errors. The risk function is also used to identify error-sensitive regions of the state space, from which additional training data can be collected. The concept is illustrated in a toy example involving pendulum dynamics and in a vision based aircraft detect and avoid application and it is shown that the collision risk is reduced by 37% over a chosen baseline.
strengths_and_weaknesses: Originality/Novelty:  The idea of systems level performance modeling and how perception errors can be linked to control for achieving particular end to end goals has been studied before. (See for instance: Greiffenhagen et al, Statistical Modeling and Performance Characterization of a Real-Time Dual Camera Surveillance System. CVPR 2000: 2335-2342.).  In this sense, the concept in the paper is well known and incremental. Other aspects including formulation of the risk sensitive loss function, dealing with differentiability issues in the loss formulation, data generation, illustration of utility in the application setting are novel.  

Reference to Past work:  A number of papers have appeared in the computer vision literature in the 90’s and the 2000’s exploiting uncertainty propagation / distribution propagation for system identification. See haralick.org for several papers on performance characterization in computer vision context. The paper does not refer to any of these past literature.  

Significance of work:  The work incorporates a risk sensitive loss function in the perception module design and is an illustration, in the modern ML (with supervised learning) context, of how error models in perception should be linked to overall risk in the control scenario.  The example case studies are narrow but sufficiently illustrate the concept.

questions: It was not clear from reading the paper if the ACAS example datasets you use are for a simulated setting or real.  How do you see the work translating to real-world settings?   How do you validate your model assumptions ?  For instance, even in this simple detection scenario, the bernoulli random variates used for probability of misdetection and probability of false alarm are complex functions of the trained yolomodel as well as the current sensing conditions (e.g. scene state, sensor noise, etc).   
limitations: It is nice to see that the authors acknowledge that the concepts presented in the paper do not provide safety guarantees.  Can you comment about what specific gaps are there from taking this work to a full blown deployable system?
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a risk driven loss and augmentation technique to improve safety performance of image based control systems. They introduce an MDP between the perception model and the controller which quantifies the risks associated with action taken at a given state, and the states which are risky. The MDP is used to quantify risk and is learned with distributional reinforcement learning. Evaluation with pendulum and a simple aircraft avoidance system shows improved results compared to vanilla perception models. 
strengths_and_weaknesses: Strengths:
- The idea is novel: use of a Q-function to assess the risk of a perception module is a departure from prior risk-sensitive systems. 
- The paper is well written. The problem statement is clear, there is sufficient background provided on the method, and the implementation details make sense. Use of inverted pendulum as an example makes the paper easy to follow.
- The evaluation with an airplane avoidance system is sufficiently complex to evaluate the hypotheses posed in the paper.

Weaknesses:
- There is no details provided on how Q-learning is done. Do you use a neural network? What are the hyper-parameters? If not a neural network, what is the computational complexity?
- The paper uses discrete systems throughout. I suspect this is because of the distributional Q-learning algorithm. The paper should mention this as a limitation and perhaps discuss methods on how it can be addressed.
- The evaluation examples are simple. Evaluation on prominent benchmarks or simulators in autonomous vehicles or other realistic control systems would make the paper more convincing. 
- Figure 2 is difficult to read, especially the line plots on the right. 
questions: I have no questions
limitations: Same as weaknesses
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a method for quantifying the control risk induced by perception errors (state-estimation errors) in partially-observable systems (e.g. image observations).
The method assumes a dynamics model and a controller are already available and focuses on providing risk awareness to the existing system.
With a predefined probabilistic model of perception errors, risk is defined as the *Conditional Value at Risk* (CVaR) of the distribution of cumulative rollout cost (i.e. cost-to-go, or return) under the uncertainty of the assumed dynamics, control policy and perception error model.
The cost-to-go distribution is estimated using a categorical distribution, discretizing the cost space.
Once the cost-to-go density is available, the CVaR risk metric can be evaluated.
It is then used as a secondary objective for training a refined version of the perception model / state estimator, as well as for active data acquisition when training that component.
The method is evaluated on the image pendulum (simulated) and on an aircraft collision avoidance data set, demonstrating significant improvement in risk-averse behavior.

strengths_and_weaknesses: **Strengths**
- Accounting for estimation errors is a good idea. Blindly using bad state estimators (what's referred to as a perception system in the paper) for MPC control can have significant negative impact on performance, so the contribution appears pragmatic.
- The method appears generic and could be applied to other settings.
- The individual ablations of the active data acquisition and the risk-sensitive loss are very appreciated.
- The running pendulum example throughout the manuscript is intuitive and provides clarity w.r.t. the different aspects of the method.
- Quantifying the relative perception risk of states, by subtracting the risk associated with a perception error of 0, seems like a useful idea.

**Weaknesses**
- The method explicitly targets partially-observable systems, with an observation space (e.g. images) that differs from the Markovian state-space (e.g. joint angles) of the system.
The ideal targetted problem should be a *Partially-Observable Markov Decision Process* (POMDP) [1], whereas the authors assume an MDP for control.
A discussion of how the method deviates from the POMDP paradigm is necessary to make its limitations clear, and is currently missing.
- For proper POMDP treatment, state estimators generally need to account for the history of observations (e.g. through probabilistic filtering), or alternatively the policy can be recursive and take the history of observations into account [1]. Tying this into my previous remark, I think this needs to be mentioned in the paper; from what I understand the considered visual perception systems map only the current observation to a deterministic state (sec. 3). The authors briefly touch on this aspect in the conclusion, but I think this deserves a more detailed discussion.
- The objective for the refined state estimator is mixing state estimation and control, because the added risk-averse term still tries to maximize the CVaR of the cumulative cost under the assumed controller.
However, the refined module is still framed as a perception system / a state estimator, which dillutes the concept in my opinion.
I would suggest positioning the trained model in another way, e.g. treating it as a sub-parameterization of the overall policy which would include the controller $g$ as well.
- The method assumes that data is available in advance and can be queried freely to optimize the risk objective or perform active data acquisition. In general, this is not the case for on-line RL, where an agent would need to query the environment consecutively and data acquisition can be expensive.
- I would have preferred more systems explored in the experimental section. The method is generic, and that would support its broad applicability.

**Further remarks**
- The overview in section 3.1 could point out what will happen with the observations to prepare the reader for what's to come. Currently it only discusses quantifying the state estimation errors, but not how that fits together with the existing state estimator. Diagram in Fig. 1b is also missing the observations alltogether.
- $Z^{\pi}$ is described as a "cost distribution function" in sec. 2, but if I interpret the equations correctly it denotes the random variable of the infinite-horizon return, for which expectations, CVaR, etc. can be computed. If my interpretation is correct, I think it could be phrased better.
- I am not convinced framing the perception error model as a policy is the best idea. The original policy is still involved in the rollout in the form of $g$, the controller, and having two policies is confusing. Why not treat it as an additive error model on top of the transition, for example?


Since the method appears generic and I like the idea of quantifying the future state estimation errors of the system, I'm leaning towards a positive assessment.
However, there are conceptual issues that need to be addressed in the manuscript and I find the evaluation of the method can be improved by considering further systems.

**References**

[1] Bertsekas, D., 2012. Dynamic programming and optimal control: Volume I (Vol. 1). Athena scientific.

questions: See above.
limitations: I don't see any major negative societal impact. Some of the method's limitations are acknowledged in the conclusion.
review_title: null

Meta-review: metareview: This paper proposes a technique that can evaluate the risk to a system (e.g., autonomous robot, aircraft, etc.) posed by errors in its perception system. While the idea of evaluating a perception system (or any other subsystem) as part of the overall pipeline is not new, this work introduces a novel method of encapsulating the risk in the cost-to-go values of the system under assumed dynamics, control policy and perception error model. The Q-values are then learnt via distributional RL. Overall, this work has been well received and should be interesting to anyone developing complicated systems which have machine learning components in them. The authors are encouraged to incorporate all the rich feedback during author-reviewer interaction period in the camera-ready version.
confidence: Certain
Decision: Accept