Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This work addresses OOD detection based on density modeling with normalizing flows (NFs). Specifically, the authors propose an approach which follows past work in modeling the distribution of logits with a NF. The authors improve on past work aiming to address topology mismatch issues between the logit distribution and base distribution. To do so, they use class conditional distributions, as opposed to a single marginal, and further reshape each class's base distribution using learned accept/reject sampling. They demonstrate their approach on object detection tasks in both simulated and real-robot OOD detection tasks.
review: Strengths:
- The proposed approach balances computational complexity and performance, yielding good density modeling yet remaining practical to deploy in real-time contexts.
- The paper is well written, and the authors provide a good amount of background on the prior advances the authors build on in this work.

Weaknesses:
- The work strikes me as a straightforward application of prior advances for improving the quality of NF density modeling to existing techniques using NFs for OOD detection. With this in mind, the main contribution of the authors is to show that such a technique works well and is efficient enough to run in real-time. Specifically, the authors claim that spending computational budget on resampling base distributions per class is better than just adding more layers to the normalizing flow. This is an interesting claim, and I would have liked to see an empirical evaluation of this tradeoff.
- The real-robot experiment was not described in sufficient detail: how were the images labeled? how were images classified as being in- or out-of-distribution? Furthermore, the real-world experiment lacked comparison to any baselines other than the base YOLO model.

Minor comments
- Some claims are contradictory:
  - Line 106-108: "This way we get similar benefits...without having to burden the cost on marginalization over classes"
  - Line 163: "We marginalize the density over classes"
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, this paper tackles an important problem and presents a solution that performs well on the domains tested, but the approach is a relatively straightforward application of existing work, and the experimental evaluation is missing useful comparisons and lacking details regarding the real-robot experiments.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper proposes a new out-of-distribution method using normalizing flows but with a class-conditional base distribution. This enables the new method to ‘topologically match’ the target distribution, thereby enabling better ODD detection. 
review: The paper proposes a novel method for ODD detection. The paper does not fully demonstrate its use in robotics. It is unclear whether the problem being addressed indeed exists in robotic inspection problems, or not. It would have helped if this was made clear right at the beginning of the paper. The paper's readability would have improved if a section on the background covered more details about the prior methods used.

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The work is good and sound. However, its use in robotics is not very clear. The robotic inspection and maintenance scenario (Sec 4.3) is not clearly explained. It would have helped the paper if either this was the key application explored, or at least two other robotic applications of the proposed ODD method were shown.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper presents a method for improving the distribution modeling accuracy of normalizing flows when the target distribution has complex topological properties, for the purpose of OOD detection in object classification from images. In particular, a fundamental limitation of normalizing flows is that the learned distribution shares the topological properties of the base distribution; hence, if the base distribution is unimodal (e.g., a Gaussian, which is commonly used), the flow may be unable to accurately model the target distribution. To overcome this, this paper proposes using a more expressive learned base distribution combined with an information bottleneck loss. At test time, the OOD metric used is the predicted likelihood, marginalized over classes. The approach leads to improvements in OOD detection accuracy on pre-existing datasets as well as on images collected on a real robot.
review: Strengths: The paper is generally clear, technically sound, and is extensively compared to baselines. Moreover, the problem of OOD detection is highly relevant for robot learning today. While the paper essentially combines existing techniques (LARS and IB) for application to class-conditioned flows, the paper could still be useful for future normalizing flow practitioners.

Weaknesses and concerns:
- As a general comment, I am curious how often multimodal distributions with pathologically disconnected supports actually occur in practice for image classification. I ask because the overall OOD accuracy improvement in the real image datasets in Table 2 seems fairly minor. For the purpose of understanding, it would strengthen the paper to analyze the learned acceptance function in these settings: does it actually learn a multimodal base distribution, like in the synthetic examples? How many disconnected modes are there? If the learned base distribution is still unimodal, it would seem that topology is not the bottleneck, but rather that using a modified based distribution may overall improve the training optimization landscape compared to using a standard Gaussian, and maybe the contribution should be stated as such.
- It is well-known that flows can often assign higher likelihood to OOD data than in-distribution data, and thus may have limited utility as an OOD measure [1]; however, this limitation is not discussed in the paper. Intuitively, this is because while maximum likelihood training encourages high likelihood values for training data, nothing constrains the training to assign low values to unseen data. Thus, flows still may be insufficient for OOD detection requirements in the safety-critical applications mentioned in the paper (one alternative is to use distance to training data and calibrated bounds on perception error, e.g., [2]).
- I would recommend putting more details on LARS in the paper (perhaps moving some details from the appendix). How is the acceptance function parameterized? How is T selected?
- In the paper, $p_{\phi,\psi}$ is used in the figure subtitles but the notation is never defined?
- Instead of LARS, what about simply using a mixture of bounded uniform distributions as a base distribution for each class? This would be potentially simpler than LARS, and could capture multimodality without inefficient sampling due to the rejection in LARS?
- It wasn’t clear to me why a class-conditional setup is required for the OOD detection problem, since in Sec 2.3, the OOD score is just computed by marginalizing over all classes anyway. Why couldn’t one just train a flow with a single class for all training images and use that to compute an OOD score for test-time images? Some discussion in the paper would help.
- In the information bottleneck loss, how is epsilon chosen, and how sensitive is the performance to the choice of epsilon?
- It would strengthen the paper to report the accuracy of the other baselines on the real robot experiment.
- For the real robot experiment, the OOD detection performance is overall much worse compared to the Pascal-VOC and MS-COCO datasets (in Figure 5c, comparing TPR for the FPRs seen in Table 2). It would be good to have discussion in the paper as to why this is; is it due to limited dataset size?
- What is considered to be an OOD image and ID image for the real robot experiment - this should be described in the results?

[1] Zhang, Goldstein, and Ranganath. Understanding Failures in Out-of-Distribution Detection with Deep Generative Models. ICML’21.

[2] Chou, Ozay, and Berenson. Safe Output Feedback Motion Planning via Learned Perception Modules and Contraction Theory. WAFR’22.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Paper is clear with extensive baseline comparisons, but could use more motivation on design choices in the method, discussion of limitations, and clarifications on the real robot experiment.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper presents a method for performing out-of-distribution (OOD) detection for trained object detection models. The authors use a Normalizing Flow to learn a class conditioned generative model of the object detector class classifier logits, and use the likelihood of this generative model to perform OOD detection. The main contributions of this paper are extending an existing work [13], which uses learned rejection sampling of the base distribution, to class conditioned base distributions. The authors also adapt the method from [18] in order to train this model with an information bottleneck (IB) objective to prevent class information being under-utilized. The authors show in experiments that their method outperforms several baselines on benchmark datasets on a variety of metrics, as well as demonstrate the method in the context of a robotic inspection task. 
review: Strengths
- This paper deals with an important issue for robot learning that is very relevant to the CoRL community
- The efficacy of the method is demonstrated convincingly by the results, with both benchmark results and a compelling real-robot deployment scenario
- I found the paper well written and the quality of exposition to be high. 

Weaknesses
- Much of the cited literature, particularly [27, 35, 45] discusses how likelihood is often a poor choice both as training objective and as a means of performing OOD detection. This supports the author's use of the IB objective for training, but the authors still use the likelihood in order to perform OOD detection. I think the authors could engage more with this aspect of the literature. For instance, did the authors try other methods for performing OOD detection such as those in [45] or [46] for instance? Why is likelihood a good metric for OOD detection in their case?
- The main contributions of the paper are an extension of [13] to class-conditioned priors, and of [18] to the resampled base distribution. The technical contributions of the paper are fairly incremental. 
- Limitations sections could be improved. Other than line 273, the limitations section is mostly justifying the method and arguing that the computational cost is negligible, rather than a frank assessment of the limitations. I would be interested to see the authors outline what tasks their current method is restricted to, or alternatively any limitations that might affect real-robot deployment. For instance, the current method does not consider the sequential nature of observations at deployment. 

Misc questions
- What is the relative increase in computational cost due to the rejection sampling vs MoG flow with the same number of flow layers? 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I believe this paper tackles an important problem, and shows promising results, though the proposed approach has limited technical novelty. The paper could be further improved with a more comprehensive limitations section and more discussion on the use of likelihood to perform OOD. On this basis I recommend a weak accept.
review_title: null

Meta-review: The paper proposes topology-matching normalizing flows for out-of-distribution detection in robot learning. The reviewers had various concerns about the approach including the novelty however it has many positives. The reviewers appreciate the authors' time and effort in addressing the rebuttal, the paper has improved. Therefore, it is my pleasure to let you know the paper is Accepted (Poster). Authors are encouraged to address the reviewer comments to further improve the paper before the publication.
Decision: Accept (Poster)