Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a method to detect OOD scenarios for robots. The method takes into account only changes that impact the robot’s performance, meaning that it neglects distribution shifts that do not affect task completion. This detection technique comes with guaranteed confidence bounds on performance, specifically on the expected cost of the policy. This bound is derived based on recently developed results on derandomized PAC-Bayes. The effectiveness of the method is shown on both simulated and real robotic experiments.

strengths_and_weaknesses: ### Strengths
The paper is very well written and structured. One can clearly see that the authors put a lot of care into defining everything thoroughly, and making sure that the readers do not have to go back and forth between sections. The experiments are also well designed and clearly organized. The results show an improvement over the baselines, which detect OOD even in cases with near-identical test expected cost (to the expected training cost), proving that the presented OOD detection is task-driven. 
The task-driven formulation also helps when the robot’s sensor observations do not include possible disturbance factors as wind strength or direction.

### Weaknesses
As already pointed out by the authors in the conclusion, when using this pipeline one still incurs the risk to fail on the test environments, which is exactly what one wants to avoid, especially when dealing with autonomous vehicles.

summary_of_recommendation: This paper presents an elegant way to detect OOD environments and training a deterministic policy with a guaranteed bound on the expected cost. When the bound is violated, it indicates that the robotic system is not able to maintain the same performance - up a threshold - on the test set.
They propose two methods which both work better than the presented baselines. The experiments on the drone also show that the environment is not considered OOD until the wind speed is increased to 75% of the maximum disturbance. The main advantage of their method is two fold:
- not detecting OOD when the cost is unaffected by the changes, so in a way, not overestimating the proportion of OOD environments
- detecting OOD in scenarios where not task-driven methods would fail, as in the wind disturbance example

For this reason, for the clarity of the exposition and for the quality of the experiments, I would recommend acceptance.

issues: To better inform future practitioners, you could give more intuition on the two methods you proposed, as in, when is better to use p-values vs concentration inequalities.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper introduces a PAC-Bayes theory-based out-of-distribution detection (OOD) framework. 
The paper proposes to train a policy whose related expected cost on the training distribution is bounded. After training the policy, OOD on the test environments can be performed with two different, proposed methods. The paper contributions are two-fold: guaranteed confidence bounds on OOD detection and sensitivity against environmental changes which matter (i.e. which changes the expected cost under the current policy). 
strengths_and_weaknesses: Strengths:

- provided methods have a bound guarantee which is proven mathematically
- paper is properly written, can be read fluently
- interesting and convincing experiments

Weaknesses:

- I don't see any main weaknesses
- Some more information regarding the policy training procedure would be nice (see Issues)
summary_of_recommendation: Even though I am not an expert in this field, I enjoyed reading the paper and learned a lot. The paper is clearly written and can be read fluently. The differences to prior work are made clear. The motivation and claims of the paper are proven via thorough mathematical derivations and experiments. 
issues: One major remark:

- I think the layout of the paper was changed, especially the space between equations and text. This should be addressed.   

I have minor points that might be interesting to know for the reader:

- It would be interesting to know if the sample complexity of the training algorithms change due to optimizing the bound/"regularized cost"

- Theorem 1 requires the training algorithm to be deterministic, as stated in the Appendix. How does this influence the training procedure? Does this mean that usage of stochastic methods is restricted? 

- What exactly is meant by transforming the training procedure into a supervised learning setting for the drone experiment?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a PAC-Bayes approach for out-of-distribution (OOD) detection within policy learning settings. The authors first apply derandomized PAC Bayes to train a policy, which provides a bound per hypothesis rather than averaged analysis over multiples hypothesis. Then, as a bound is provided per hypothesis, the paper proposes to check the violation of the PAC-Bayes guarantees at test time, i.e. if the PAC Bayes bound is violated, the authors attribute this to OOD scenario. Concretely, this is achieved via hypothesis testing and checking the confidence intervals. Theoretically, a bound is established for these two schemes, which holds within a pre-specified interval. The method is task driven, as the expected costs are used. Experiments with 2 simulations and 1 real robot - in particular, the drone performing vision-based obstacle avoidance - the authors validate the approach. Overall, the paper provides an interesting application of PAC-Bayes to OOD detection within the policy learning settings.
strengths_and_weaknesses: 

Pros:

- To my knowledge, the proposed application of PAC-Bayes to OOD detection within the policy learning settings is novel and interesting. Some of the derivation steps that the authors demonstrate, are non-trivial and thought involving.

- The paper's focus on "task-driven" OOD detection - only figuring out the OOD samples that affect the performance of the policy - is relevant to safety-critical settings of robot learning, and the paper shows real robot experiments as well as some of the theoretical foundations associated with their method. This is likely to be appreciated at CoRL. 

Cons

- The paper has several rooms for improving the exposition.

First, the margins seem to be somehow reduced, when compare to other CoRL submissions. This can be checked and corrected if inappropriate, during the rebuttal period. Second, while the insights are very clear, it might help to provide some remarks at the end of each theorems - ideally, in a common tongue or insights. Third, for the proofs in the Appendix, it might help to first state the proof paths, and provide the proofs, so that the work is more accessible. For the video, you may want to make it standalone. Currently, it is difficult to get the relevant information before reading the paper. For example, we see the drone crash at OOD settings, but there are no explanations.There are also minor comments which I list at the end of the review.

- The comparisons to the existing approaches are kept minimal when compared to rational works on OOD detection. A compelling comparisons to other OOD methods can further show-case the strength of PAC-Bayes approaches within the considered scenario.

To back up, the paper compares to MSP - iclr 2017 and MaxLogit - arxiv 2019 (figure 3), which are both targeted for general settings, and it is unclear whether MSP and MaxLogit are the state-of-the-art alternatives. The paper should demonstrate how far the idea has advanced the state-of-the-art OOD detection literature, e.g. compared to other PAC-Bayes OOD detection methods (ref 14-16), as well as other task-driven OOD methods such as ref 18-22.  

The reason for emphasizing the comparison to the state-of-the-art is that the provided bounds are statistical, and there are chances that the guarantees are violated. An interesting question is then, how meaningful it is to have such guarantees, when the resulting failures are catastrophic.
summary_of_recommendation: I recommend accepting this paper. The paper provides principled approach to OOD detection using PAC-Bayes bounds. It seems original, and technical quality is good - the approach is reasonably backed up by the existing theory, and the real robot experiments show-case the success of their method, which is likely to be appreciated at CoRL.

My main concern is on the clarity of presentation (which is good but still has rooms to improve - the bars are usually high with theory) and lack of rigorous comparisons to the state-of-the-art alternatives.
issues: - eq (4) §\hat{S}§ seems undefined.

- eq (2), eq (3), eq (5), eq (6): P is undefined (though standard in PAC-Bayes, may not be for OOD detection communities).

- figure 2 & 3: indicate what is better or worse, or what the data means intuitively.

- other comments on writing above.

- more comparisons to the state-of-the-art.

- A.17 to A.20 and A.27 to A.30 needs more explanations.

- figure 3 missing the plots of varying environmental difficulties and only shows wind conditions variations.

- setting m bigger than 8, how does this influence the derandomized PAC Bayes? At testing, do you need a batch as well?

- The paper trains a policy with derandomized PAC Bayes. Would this affect the performance of the trained policy over the previous PAC-Bayes policies OR even the state-of-the-art methods? How difficult is it to train as well? Also, how tight is the bound in practice? 

- In the appendix, the authors refer to different sources. For the researchers working on OOD detection, providing more background materials in the appendix can help in understanding the paper. For the researchers familiar with PAC-Bayes methods, it can help in not resorting to other papers. 

- An empirical validation of the proposed statistical guarantees can be meaningful, e.g. histogram analysis?


-----------------------------------------------------------------------------------------------------------------------------------------------

Post Rebuttal

-----------------------------------------------------------------------------------------------------------------------------------------------

I thank the authors for the detailed response. I have thoroughly read the comments from other reviewers, the author response and the revised manuscript. I am happy to see that the revised manuscript has improved in its clarity.

However, I maintain my score; I find the paper to be a borderline due to the following reasons:

-  (On significance) more informative comparison to the state-of-the-art.

The provided use-case of the method is clearly on OOD detection for the policy learning, and yet, the advancements to the OOD literature is unclear.

My first point is on the unclear motivation behind the chosen baselines. As evidenced by Section 2, the authors state that the missing gaps in "Anomaly/OOD detection in supervised learning" is the need for the specific network outputs, e.g. softmax (classification) and perhaps covariance (regression). However, the authors still compare to the approaches for supervised learning: MSP and MaxLogit. Therefore, I found the motivation behind the chosen baselines unclear.

My second point is the need to choose other "task-driven OOD detection approaches" as the baselines. As evidenced by Section 2, the authors discuss other "task-driven OOD detection" approaches, and point out that the missing gaps are the statistical guarantees. However, the advancements to the literature is unclear as the authors do not provide any comparison to other "task-driven OOD detection" approaches. Therefore, I found the advancements to the state-of-the-art unclear.

While the authors address my concerns to an extent, I still found the arguments not fully convincing. For example, the argument that MaxLogit (in 2017) has shown to outperform other alternatives is not convincing. This is because the settings of the experiments are different, and the comparisons of MaxLogit are limited to the approaches prior to 2017. Moreover, the reason behind not comparing to other "task-driven OOD detection" approaches is because the proposed method requires the "roll-out" in some test environments, which is the limitation of the proposed method over the existing methods. In this regard, the readers could appreciate more if the authors have resolved this limitation, and provided a comparison to the other task-driven OOD detection methods.

This is the reason why I rate the significance of the paper low; the paper doesnt have to outperform the other approaches, but should contain the required information for the readers to illustrate the advancements to the current approaches.

-  (On significance) the required roll-outs in test environments.

Taking more global view, many existing approaches do not require the roll-outs in the test environments, which is the main point for developing OOD detectors (targeting at safety-critical applications). Simple examples are the baselines chosen by the authors - MSP [3] and MaxLogit [5] (and many others, e.g. reference [4, 17]). Therefore, my question still remains: how meaningful it is to have the provided method with some guarantees, when the resulting failures are catastrophic (over the existing task-driven methods despite the given drawback of the approach).

-  (On thoroughness) the analysis of the batch-size at testing.

The proposed method can be sensitive to the chosen batch-size at testing (as the authors provide the information). Then, perhaps, an ablation study that guides the choice of an optimal batch-size would be meaningful.

-  (On thoroughness) more data samples for varying environment difficulty.

Only 2 evaluation points for varying environment difficulty might also be limited in thoroughness, as oppose to more data points with wind variations. The paper could improve with more data samples for varying environment difficulty.


-----------------------------------------------------------------------------------------------------------------------------------------------

Post Rebuttal

-----------------------------------------------------------------------------------------------------------------------------------------------
review_title: null

Meta-review: The paper proposes a new technique for detecting when the robot is operating in an environment that is different from the one it trained in. This problem is known as out-of-distribution (OOD) detection, and it occurs often in robot learning, especially when new elements (such as wind) are introduced, or when a system is trained in simulation and transferred to the real world. Interestingly, the proposed method does not detect changes that do not affect the performance of the robot. The central idea behind this work is to train the robot with PAC-Bayes performance guarantees. When the robot operates in a given environment, it continuously measures its own performance and compares it against the predicted theoretical performance. When the the difference between the two is sufficiently large, based on p-values of hypothesis testing, then an OOD is detected. The proposed method is evaluated on 2 tasks in simulation and one real robotic task.
The reviewers agree that the technical contribution of this work is solid and novel. The paper is over all well written, although several technical details are not clear. It seems like the paper is written while assuming that the readers are all familiar with the PAC-Bayes literature and notations. Another issue raised by a reviewer is the utility of this approach in practice. It seems like OOD detection after a failure occurs is too late for sensitive robotic applications, such as self-driving cars. In these application, OOD should be tackled before a drop in the performance of the robot, contrary to the central idea of the proposed method here.
Decision: Accept (Poster)