Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an approach for detecting out of distribution (OOD) samples by synthesizing outlier samples and considering an unknown-aware learning mechanism. The synthetic outlier samples are used to learn a tighter class boundary for in-distribution samples. The model is trained with an uncertainty-aware loss function that encourages a high probability score in distribution samples and a low probability score for OOD samples. 

main_review: Strengths:

1. Considering synthesized OOD sample to expose model is more feasible compared to procuring real OOD samples. Authors synthesis the OOD samples in the low-dimensional feature space which is more efficient to generate than the actual image space. 

2. Unknown-aware training is also an effective idea to differentiate the in distribution samples from OOD samples.

3.  Experiments are conducted on the benchmark dataset and results are impressive. Extensive experiments and ablation studies are conducted to evaluate various components of the proposed approach.

Weaknesses:

1. Overall, the novelty of the proposed approach is limited. Considering synthesized outliers to detect OOD or novel objects are explored in various previous approaches. Authors consider a multimodal Gaussian distribution to approximate in distribution samples and sample new OOD samples. However, the class boundaries in the high-dimensional space may not be Gaussian.

2. Authors consider the object detector network for generating the bounding boxes for the OOD datasets. In this case, the object detector may miss a significant number of bounding boxes as some of the objects instances may not appear in the in-distribution data. In this case, these boxes will not appear for the OOD test. Do authors see this behavior and have any comments on this issue.

3. In table 2, while comparing the other generative methods, do authors consider the same method for detecting OOD. For example, if the additional loss function is included in the approach, then the comparison may not reflect just the benefit of the proposed synthesis method. 

4. In table 1, authors are also encouraged to provide the backbone networks for the methods they compare against so that contribution can be better evaluated. 

5. What is the motivation for considering the synthetic OOD samples when real outlier data are abundant, easily available, and achieve better performance. 
summary_of_the_review: The authors proposed an OOD detection approach the relies on synthetic outlier samples. The proposed approach is shown to be effective but the overall novelty is limited.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The manuscript addresses open-set object detection by modifying the ROI classifier of Faster-RCNN. The probability p(O|x,b) is modeled as sigmoidal activation of a generalized energy score of the classification head logits. The method succeeds if i) RPN ignores the outlier object, or ii) RPN detects the outlier but the candidate gets rejected due to high p(O|x,b).

The generalized energy is formulated as log-sum-weighted-exp-logit in equation (6). The probability p(outlier|x) is modeled as a sigmoid of modulated generalized energy in equations (8) and (9). The classification head is trained to output high energy in inliers and low energy in artificial outliers through equation (8). The artificial outliers are sampled from low-likelihood regions of per-class Gaussians which are fitted to latent features of groundtruth ROIs. These Gaussians are periodically updated during training. 

It appears that the authors consider ROI-wide representations of RPN candidates. These representations are likely produced by applying ROI-align and two fully-connected layers to the convolutional features of the RPN. These Gaussian distributions are modelled according to per-class means (\mu_k) i and overall covariance (\Sigma). Virtual outliers are sampled so that their likelihood with respect to \mu_k and \Sigma is less than \epsilon. 

main_review: Strengths:

S1. Sampling artificial training samples in the feature space appears effective.

S2. There is some novelty in the formulations of uncertainty score (learned weights) and outlier probability (sigmoid of modulated uncertainty score).

S3. The authors took time to implement and evaluate several prominent baselines. Experimental evaluation shows competitive performance.

Weaknesses:

W1. Reading the paper requires a lot of guesswork. Figure 2 suggests that the outliers are detected within RPN while in fact the paper considers only ROI classifier (it would be helpful to indicate the location of ROI-align and others layer of the ROI classifier). It is not clear why \Sigma is used instead of \Sigma_k. 

W2. The paper does not discuss the computational complexity of recovering \Sigma. 

W3. The following work should be cited and discussed since it considers logit normalization for outlier detection:
[R0] Sanghun Jung, Jungsoo Lee, Daehoon Gwak, Sungha Choi, Jaegul Choo. Standardized Max Logits: A Simple yet Effective Approach for Identifying Unexpected Road Obstacles in Urban-Scene Segmentation. ICCV 2021.

W4. The following work should be cited and discussed since it considers learning open-set recognition with artificial outliers:
[R1] Zhi-Lin Zhao, Longbing Cao, Kun-Yu Lin. Revealing Distributional Vulnerability of Explicit Discriminators by Implicit Generators. CoRR abs/2108.09976 (2021)
[R2] Matej Grcic, Petra Bevandic, Sinisa Segvic. Dense Open-set Recognition with Synthetic Outliers Generated by Real NVP. VISIGRAPP (4: VISAPP) 2021: 133-143

Suggestions
- Equation (8) could be clarified by using the notation from (9).
- Would it make sense to include OOD AP in Table 1?
- Propose some intuition on why the present method outperforms [liu20nips] in Table 3. Is it only due to w_k?
- Quantify how many OOD objects are ignored by RPN-a, rejected by (10) and detected as in-distribution objects.

Post rebuttal comment

The revision improves the presentation and clarifies some doubts. Hence I propose to raise my rating to accept.
summary_of_the_review: The manuscript proposes a reasonable baseline for open-set object detection.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed a novel unknown-aware learning framework dubbed VOS (Virtual Outlier Synthesis), which optimizes the dual objectives of both ID task and OOD detection performance. The key of VOS is to sample outliers in low-dimension feature space instead of generated images. These outliers are used in the training process to help form the compact decision boundary between ID and OOD data.

The extensive experiments show the effectiveness of their method on both the PASCAL-VOC and the DeepDrive-100K datasets. 
main_review: Strengths:
(1) This paper is well written and easy to understand. Especially,  the figures deliver the core idea of the whole paper.
(2) The proposed method is straightforward but efficient. Also, the idea could be proved by a toy example.

Weakness:
(1) Some extreme cases are not clearly illustrated. For example, the sampling process is done online. In one batch, it is likely to have only one instance in some class. In that case, I am very concerned about the quality of estimation.
(2) Too many ablations studies are arranged on supplementary files. 

Also, I have a question about the unknown-aware training objective. In this part, I understand it like a binary-cross entropy with learnable weight. I am wondering about the influence of learnable weight. I checked the visualization in the supplementary file but do not find the performance effect.


summary_of_the_review: Overall, this paper could clear deliver the idea to the readers with well-written text and designed figures. The idea is somewhat novelty by the related work. The unknown-aware training object is similar to the dynamic focal loss with the virtual outlier. I agree that the motivation about sampling virtual outliers in the feature space instead of pixel space. Therefore, I vote this paper marginally above the acceptance threshold.

Given that I am not very familiar with OOD, I would like to check other reviewers' opinions.

——————————After Rebutal ——————————

The author has addressed most of my concerns. In such a situation, I would like to raise my scores.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an effective method for OOD detection and model regularization, which does not rely on real OOD datasets. Specifically, the proposed method synthesizes virtual outliers by sampling low-likelihood samples in the feature space of class conditional distribution, and adds a novel regularizer to the original ID training objective, which contrastively shapes the uncertainty space between the ID data and synthesized outlier data. Extensive and comprehensive experiments demonstrate the effectiveness of the proposed method.
main_review: Strengths:
1. This work looks solid, and the experiments in this paper are sufficient, which consider almost all key parameters or components of the proposed method and demonstrate the effectiveness of the method. Besides, the method achieves state-of-the-art performance on both object detection and image classification tasks.
2. This paper is well motivated, which aims to utilize synthetic virtual outliers to regularize the model, thus avoiding the collection and cleaning of real OOD data.

Weaknesses:
A few details in the proposed method are not very clear. Specifically, what are the motivation and mechanism of the proposed generalized energy score? How to train the learnable weight coefficients? How to update the ID queue? It would be better to clarify these issues.

Other concerns:
1. Since the real outlier data may not be tightly distributed around the ID data in raw or feature space, why the performance of OE can be viewed as a performance upper bound of VOS? 
2. I wonder what virtual outliers look like. It might be better to visualize them.
summary_of_the_review: Overall, this paper is technically and experimentally sound, which conduct comprehensive experiments to verify the proposed method. The reasons to accept outweigh the reasons to reject (e.g., a few unclear details).
review_title: null

Meta-review: This paper proposes to synthetize virtual outliers by sampling from low-likelihood regions of the feature space of a class conditional distribution, in order to make more robust predictions via a regularization loss term.

In the reviewing phase certain criticisms were raised by reviewers: namely that i) the paper was not clear w.r.t. its goal, motivation and position in the literature of OOD detection for bounding boxes; ii) details about the energy-based formulation and covariance definitions and iii) experimental setting and metric used were missing. During rebuttal the authors answered to all the above criticisms up to a satisfying extent and were able to increase two reviewers' scores.

The paper is accepted conditioned on the fact that the camera-ready includes the additional details and discussions that arose in the comments with a specific emphasis on properly framing (and limiting) the motivation of OOD detection for open-set object detection and it is expected to properly cite the literature of the more general OOD detection task as discussed in the comments.
Decision: Accept (Poster)