Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper attempts to solve OOD problem in object detection.
It tries to reduce the impact of lacking unknown data for supervision and leverage in-distribution data to improve the model’s discrimination ability with both Information Bottleneck and Reverse Information Bottleneck.
It first uses a standard IB network to disentangle instance representations that are beneficial for localizing and recognizing objects, and then use a RIB to obtain simulative OOD features to alleviate the impact of lacking unknown data.
Experiments demonstrate good performance in incremental object detection. 
strength_and_weaknesses: Strength
+ The motivation is clear, and the solution is novel.
+ Good performance gain on incremental object detection setup.

Weakness
- The major concern is the scalability of such method. The author only uses PascalVOC and BDD as in-domain data, which is relatively small and simple. To verify its ability of handling real-world data, I would suggest the author can demonstrate a more complex setup, for example, use COCO as ID and Openimages as evaluation set.
- Figure 2 is not very illustrative. It would be better to align equations into the flow. Or teardown to multiple small figures for better clarity.
- In Figure 4, the visualization of OOD is confusing. It is more like white noise. How does simulate OOD feature extracted from it help?
clarity,_quality,_novelty_and_reproducibility: The paper is well written, but packed with equations. It would be better to introduce more figures to increase the clarity.
summary_of_the_review: Overall, this is a solid paper with detailed experiments and visualizations. The major concern is the scalability in really-world data. I will increase score if the author can provide my evidence.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: - This paper proposed a new information theoretic approach to tackle out-of-distribution object detection (OOD-OD). 
- Authors leverage Variational information bottleneck (IB) to suppress object-irrelevant representations to improve discriminability of in-distribution class (ID). 
- Next, to simulate OOD features to train an OOD detector in an unsupervised fashion, authors reverse the IB equation, called Reverse Information Bottleneck (RIB). In contrast to IB, which aims to extract task-relevant compact codewords/features, the goal of RIB is to simulate task-irrelevant OOD style features to train a strong OOD detector. 
- Extensive experiments on standard OOD-OD datasets and incremental object detection datasets show the effectiveness of the proposed approach for OOD-OD.
strength_and_weaknesses: Strengths
- The proposed method is very simple and effective and is one of the major strengths of this work.
- The use of information theoretic methods, the IB, for object detection is new and fresh.
- All the claims made in the paper have been properly addressed with adequate experiments.
- The paper tackles a very important practical problem with object detectors and achieves impressive performance on standard benchmarks.

Weaknesses
- I believe the comparisons in Table-1 are unfair as the current method introduces more trainable parameters than VOS. Not to diminish the impressive performance, I believe it is imperative to add a column with backbones and trainable parameters introduced for a more fair comparison.
- Authors claim that the OOD map simulates outlier samples and the visualizations of the map doesn't seem to do that. After careful observation one can notice that the maximum of the OOD map still occurs either inside or around the object of interest. Additionally, these maps are not interpretable, it is not clear to me why these maps should look like noise. Can these maps be interpreted as adversarial noise (on the features) since they are obtained by maximizing the discrepancy between logits of the right class? (Like noise that can be added to make the feature lie closer to a different ID class different than the one in the feature map).
- In the standard OOD-OD setup the number of OOD samples coming from COCO/Open-Images are usually less than 20% (930 OD samples for COCO with 4952 samples of ID val set). It is not clear how the proposed model does with higher levels of contamination from OOD classes. 
- The contribution of the first stage of IB is not clear from the ablation experiments. What happens if one were to simply pass F through two different convolutional blocks and keep everything the same? In Table 4, authors either keep the full branch or remove it entirely, but why do one need the first IB/RIB stage? One can apply the same L_dis, L_uncertainty losses on this architecture even without computing mean and standard deviation and combining them. I understand that the loss becomes uninterpretable but how does it fare against the proposed approach?

Minor questions/comments to improve the paper (Not dependent on the final decision):
- It is not clear how Eq. 5 is related to Variational Information Bottleneck. From the paper there should be a negated term and the second term is the KL divergence. In Eq. 5 there are two KL divergence terms and it is not clear how the authors end up with this equation. This step requires some more information to help the readers understand the math.
- How is Eq. 8 related to Eq. 7. Do the authors mean to say that Eq. 8 approximates Eq. 7. In that case why is it hard to compute Eq. 7 directly without any approximation?
- It would be beneficial to researchers implementing this work, if information about computing mutual information (I(.,.)) on 3d feature maps is provided in the supplementary. 
- Is the purpose of experiments on Incremental object detection just to show that IB can improve overall object detection performance?. If uncertainty loss is not computed for these experiments, then why would one need to perform RIB?
- Kindly add the baseline to Table 4, readers would have to move back and forth to understand what the baseline is. 
- Line before "Metrics" in Section 4.1 ".... manually examined the OOD images to ensure .." -> ".... manually examined to ensure ... ".
- What is the confidence threshold used to generate the results in Fig. 3. If its the same value for VOS and TIB?

clarity,_quality,_novelty_and_reproducibility: -The writing clarity is really good and the idea of leveraging information bottleneck to enhance object relevant information and suppress object irrelevant information is novel. 
- With a few additional details asked above being answered, I believe a skilled researcher/engineer should be able to reproduce this work.
-  I believe with a few minor revisions, the paper is of great value to the conference.
summary_of_the_review: - Overall I'm very happy with the paper and its writing quality. 
- Authors leverage Information bottleneck to simultaneously enhance object relatedness and suppress (with reversal) object irrelevance in features to improve OOD-OD. 
- The proposed method is simple and very effective. The experiments support all the claims made by the authors. A few more explanations to understand the working of Reverse IB are required which I believe the authors can provide in reasonable time. 
- I'm willing to improve my ratings depending on the author's response.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the problem of out-of-distribution object detection (OOD-OD), where an object detector has to detect unknown objects without relying on an auxiliary dataset. The authors propose a framework of Two-Stream Information Bottleneck (TIB), with a standard IB network to extract object-related information from in-distribution (ID) data, and a Reverse Information Bottleneck (RIB) to obtain OOD features. By exploiting IB, the method manages to obtain enhanced object-related information from ID data. At the same time, TIB obtains simulative OOD features by maximizing the discrepancy with the ID features. Experimental results show the method could significantly improve the performance of OOD-OD and incremental object detection over the baselines.
strength_and_weaknesses: **Strengths**
+ The idea of using IB to extract object-related and OOD feature problems is novel and interesting. It seems that the idea could be applied to other object detection problems. The empirical results are encouraging.
+ The paper is well-written and easy to follow. The key design choices in the paper are carefully described.

**Weaknesses**
+ The task of OOD-OD is not formally described in the paper.
+ Does background differ from OOD instances?
+ Is the OOD bounding box included when performing inference?
+ The correlation between Information Bottleneck and the method is not clearly illustrated. Particularly, the KL terms in Eq. (5) do not express the IB where Z should have low mutual information with F but ignore the high mutual information with the label. So do the Q and P_in. Eq. (8) also has that problem. Also, we usually minimize a loss (with a non-negative value) not maximize a loss as in Eq. (8).
+ The enhancing process (Eq. 4) appears to be unclear. Both A and α are related to P_in, and the role of each term is not well described. An ablation study about this process may help to clarify.
+ Regarding the datasets, how many classes are in the ID and OOD data?
+ Images in Figure 4 only contain objects from ID classes, which makes the visualization of OOD maps uninformative. What do the OOD maps look like with objects from  OOD classes, and in the cases of mixed ID and OOD classes? It would definitely demonstrate the effectiveness of the proposed approach on OOD-OD.
+ There is no discussion about the choice of hyper-parameters in Section A.3. The paper should include an explanation for each set of hyper-parameters.

clarity,_quality,_novelty_and_reproducibility: + The paper's structure is generally well-organized. The problem and the motivation are well described. However, there are some problems with the clarity and quality, i.e., the authors use many notations without clearly summarizing or referring back to Figure 2. It would be much better to follow if the authors include a summarized table of notations and add the notation, e.g. F, P_in​, E, ….  to Figure 2.
+ A limitation section should be included.
+ The method of applying IB to out-of-distribution appears to be novel. The designs of TIB and RIB could also be useful to the community since they could be easily applied to other object detection problems.

summary_of_the_review: The idea in the paper is well-motivated and described. The paper addresses the problem of out-of-distribution object detection by exploiting Information Bottleneck, which has been carefully studied by prior work and shown promising results. Therefore, the idea has solid theoretical reasoning. However, when implementing the IB in terms of loss functions, not all properties are guaranteed. Also, lacking of a visual explanation of the OOD map with OOD instances significantly reduces the convincingness of the proposed approach.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies a crucial problem of out-of-distribution object detection (OOD-OD). The goal is to detect test-time objects that are outside the known training categories, while at the same time ensuring accurate localization and classification of in-distribution objects. To tackle the problem, this paper introduces a new information-theoretic approach to synthesizing OOD feature maps, which are then used to regularize the training of the object detection network. The novelty behind generating OOD features is based on Reverse Information Bottleneck (RIB), which promotes task-irrelevant information. All reviewers recognize the novelty of the paper, which first connects IB to OOD-OD.

Another strength of the paper is the extensive evaluation and promising performance on the OOD-OD benchmark, for both PASCAL-VOS and BDD datasets as in-distribution. The authors further conducted evaluations on the incremental object recognition task. It might be useful to report results across multiple runs with different random seeds as well. 

During the discussion phase, the authors addressed several concerns and improved the clarity of the paper. Some reviewers questioned the soundness of the approach. Specifically, there are two major points raised:

* The gap between mutual information theory and the actual realization in this paper. (the authors have made an attempt to clarify this, with additional experiments)
* The lack of clear insight and evidence of why the proposed method works. In particular, all three reviewers brought up the question of how those obtained OOD feature maps improve the OOD detection performance. To fully understand this, I believe there are two separate analyses needed: (1) The generated OOD map deviates from the instance features, and (2) why training the object detection model with the noise-like OOD feature maps can effectively generalize the detection to future unseen OOD objects, most of which may not necessarily have noise-like patterns. While the authors have provided evidence for (1), but the crucial analysis for (2) is currently missing. Instead of learning the noisy OOD feature maps using RIB, an interesting baseline would be to directly generate noise (with similar magnitude) for performance comparison. It would be important to understand the characteristics of the learned OOD feature maps and how they help with the detection of real OOD objects. A related suggestion made by Reviewer ajPF would be meaningful to include in the revised manuscript as well. In particular, _``what do the OOD maps look like with objects from OOD classes, and in the cases of mixed ID and OOD classes?''_

Overall the reviewers think the paper is in reasonable shape, but the second point raised above does cast technical doubt on the readiness for ICLR. The manuscript will be more convincing after addressing this important concern. 

justification_for_why_not_higher_score: There are some major technical concerns that are not fully addressed in the current manuscript. Please see the meta-review for details. 
justification_for_why_not_lower_score: NA
Decision: Reject