Zero-shot Object-level Out-of-distribution Detection with Context-aware Inpainting

Authors: Quang-Huy Nguyen, Jin Peng Zhou, Zhenzhen Liu, Khanh-Huyen Bui, Kilian Q Weinberger, Wei-Lun Chao, Dung D. Le
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: out-of-distribution detection, zero-shot, generative model

Abstract:
Detecting when an object detector predicts wrongly, for example, misrecognizing an out-of-distribution (ODD) unseen object as a seen one, is crucial to ensure the model’s trustworthiness. Modern object detectors are known to be overly confident, making it hard to rely solely on their responses to detect error cases. We therefore investigate the use of an auxiliary model for the rescue. Specifically, we leverage an off-the-shelf text-to-image generative model (e.g., Stable Diffusion), whose training objective is different from discriminative models. We surmise such a discrepancy would allow us to use their inconsistency as an error indicator. Concretely, given a detected object box and the predicted class label, we perform class-conditioned inpainting on the box-removed image. When the predicted object label is incorrect, the inpainted image is doomed to deviate from the original one, making the reconstruction error an effective recognition error indicator, especially on misclassified OOD samples. Extensive experiments demonstrate that our approach consistently outperforms prior zero-shot and non-zero-shot OOD detection approaches.

Decision: reject
Reviewer scores: 5, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730066413626

thread_type:
official_review

<thread_object>
creator: reviewer_h9zl

<content>
summary:
This paper addresses zero-shot out-of-distribution (OOD) detection in object detection. Since object detection is often overconfident, the OOD scores computed using only the confidence values of object detection are often unreliable. 

The proposed method uses text-based image inpainting with Stable Diffusion; The proposed method first input the original image in which the regions of the detected bounding box are masked and the name fo the detected object class to Stable Diffusion to get the synthesized image in which the masked regions are inpainted. OOD detection is performed based on the similarity of the embeddings between the inpainted and the original images.

Experiments using PascalVOC and BDD100k as ID datasets and MS-COCO and OpenImages as OOD datasets show the superiority of the proposed method over several existing OOD detection methods.

strengths:
- The idea of detecting OODs by the similarity of the images before and after inpainting is general, simple and interesting.

- The method of achieving inpainting that preserves the original context associated with the OOD classes by masking the areas that are r% smaller than the bounding box is also simple and interesting.

- The experiments and analysis are generally comprehensive, although there are some serious shortcomings, as discussed below.Although somewhat artificial, the analysis to improve misclassification of ID classes is also interesting.

- The paper is generally well organized and easy to follow.

weaknesses:
**W1.** The idea of using image inpainting for OOD detection is novel in itself, but its technical novelty is somewhat limited because it is basically a combination of existing pre-trained image inpainting and classifiers.


**W2.** OOD detection using image inpainting, while interesting, yields several weaknesses.

*W2-1.* As the authors themselves discuss, high-quality inpainting is computationally time-consuming, as inference also takes a long time for each image. This can be a drawback since object detection has many applications where processing time is crucial.

*W2-2.* If the ID class and OOD class are close, image inpainting with clear class distinctions and precise similarity evaluation may become difficult, resulting in less accurate OOD detection. It would be good to evaluate detection performance on datasets that include fine-grained categories and attributes, for example, LVIS and FG-OVD. This would also be relevant to the discussion around Line 076. It would also be good to discuss the object categories that are difficult to inpaint and their impact on performance.

*W2-3.* The results of inpainting are dependent on the initial noise, which can lead to large variations in OOD detection performance. Discussion on this point would be appreciated.


**W3.** Comparisons with existing methods are somewhat lacking. In this paper, the problem is defined as the task of binary classification of each detected object bbox as to whether it is ID or OOD. According to the task definition, a simple baseline would be to apply a zero-shot OOD detection method to each bbox. In particular, the proposed method uses CLIP, but there are other CLIP-based zero-shot OOD detection methods (e.g., CLIPN [Wang et al., ICCV'23]) and few-shot OOD detection methods (e.g., LoCoOp [Miyai et al., NeurIPS'23] and NegPrompt [Li et al., CVPR'24]). Comparisons with these methods should be included.


**W4.** The analysis presented in Table 4 is not exhaustive and does not fully support the validity of the triplet similarity in Eq. 4. In general, adding more hyperparameters should help improve accuracy. To demonstrate the validity of Equation 4, it is necessary to show exhaustive results for sensitivity to alpha and beta, with similarity(ori, \hat{y}) and similarity(ori, inp) each showing performance improvement over a wide range of alpha and beta.


**W5.** Modern object detectors are often open-vocabulary. Demonstrating that the proposed method is effective even in open-vocabulary scenarios is preferable.


**W6.** Other minor points

- RONIN (the name of the proposed method) first appears in the caption of Fig. 1 and Line 059 in the introduction section, but without saying what it stands for. There should be the definition.

- Bold b's appear in Lines 119 and 120 should be italic.

- Fig. 1 and Fig. 2 have considerable overlap in information and could be combined into one.

questions:
Since the use of image inpainting is the most important idea of the proposed method, its strengths and weaknesses should be fully discussed. In this regard, I would expect thorough responses on **W2**. I would also like to see answers to the lack of comparison with existing methods (**W3**) and the lack of analysis (**W4**), as these are critical issues to clarify the validity and effectiveness of the proposed method. Given the recent emphasis on open vocabulary scenarios for object detection, I would expect an answer for **W5** as well.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730369984606

thread_type:
official_review

<thread_object>
creator: reviewer_os8e

<content>
summary:
This paper proposes a new object-level OOD detection method called RONIN, which leverages the off-the-shelf diffusion model to replace detected objects with inpainting.  RONIN adopts a context-aware class-wise inpainting method and combinations of similarity assessment methods. Experimental results show that the proposed method outperforms existing methods in three of 4 settings on Pascal-VOC and BDD-100k.

strengths:
- S1. Using diffusion inpainting strategy for object-level OOD detection is novel. 

- S2. The proposed method outperforms existing methods in three of four settings.   

- S3. This paper is well written.

weaknesses:
- W1. The inference times of comparison methods are not reported. I am concerned that this method may have a longer inference time than the comparison methods. Since the actual application of object-level OOD detection is in areas that require real-time inference, such as autonomous driving, I believe it is necessary to accurately report the inference time of comparison methods.

- W2. I wonder the reason why RONIN’s performance is lower than the comparison methods in BDD-100k vs. OpenImages. It might be better to include the reason for this result. This is because BDD-100k is a driving dataset and closer to the real-world application than VOC.

questions:
I would like to know the inference time of the comparison methods.  

I also wonder the reason why RONIN’s performance is lower than the comparison methods in BDD-100k vs. OpenImages.


I am willing to raise the score, taking into account discussions with the author and the opinions of other reviewers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730536635830

thread_type:
official_review

<thread_object>
creator: reviewer_9vvi

<content>
summary:
This paper introduces a zero-shot object-level out-of-distribution detection method using context-aware inpainting to improve the model's ability to identify unseen objects. By leveraging generative models for class-conditioned inpainting, the differences between detected and original objects serve as indicators of recognition errors. Experiments demonstrate that this approach outperforms existing zero-shot and non-zero-shot OOD detection methods.

strengths:
1. Proposes a novel approach by integrating generative models for OOD detection, offering a solution distinct from traditional methods.
2.  Supported by data, showing exceptional performance across multiple benchmark datasets.
3. The method does not require modifications to pre-trained models, facilitating integration into existing systems.

weaknesses:
1. High reliance on generative models may be limiting, especially if the models are inaccurate or under-trained.
2. The need for extensive image generation and comparison could lead to high resource consumption.
3. Further validation on a variety of datasets is needed to ensure broad applicability and robustness.

questions:
Please refer to the Weaknesses box.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730573277681

thread_type:
official_review

<thread_object>
creator: reviewer_ohfd

<content>
summary:
This paper presents RONIN for object-level out-of-distribution (OOD) detection. Given a detected object box and its predicted class label, RONIN first performs class-conditioned inpainting on the image with the object box removed. It then compares the similarities between the original detected objects and their inpainted versions to identify OOD instances.

strengths:
1. The concept of using inpainting and comparing similarity scores for OOD detection is both reasonable and straightforward.
2. The proposed method outperforms zero-shot baseline methods by noticeable margins.

weaknesses:
1. The technical contribution is limited. The proposed method simply combines image inpainting with similarity comparison, both of which are straightforward and well-established techniques. Overall, the original technical contribution is insufficient.
2. The paper lacks comparisons with recent methods. The latest method cited in Table 1 is from 2022. Comparisons with more recent approaches, such as [1] and [2], are expected.
3. It seems from Table 4 the results are highly sensitive to the choice of $\alpha$ and $\beta$, suggesting that careful parameter fine-tuning may be required.

[1] Li et al. "Learning Transferable Negative Prompts for Out-of-Distribution Detection." CVPR 24

[2] Wang et al. "CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say No." ICCV 23

questions:
1. The authors did not conduct experiments on ImageNet-1K, a commonly used dataset for out-of-distribution (OOD) detection. Could the authors provide results on this dataset as well?
2. Instead of performing image inpainting, would generating a complete image of the predicted category using a pre-trained diffusion model work? Can the authors include comparisons with this simple baseline?
3. In Line 261, it is mentioned that "one can use a thresholding method to identify OOD objects effectively". What is the threshold used in the experiments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732318997818

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank all the reviewers for their time and valuable comments. We appreciate that the reviewers found our approach novel (reviewer 9VVi, os8e), our performance strong (reviewer oHFd, 9VVi, os8e), and our paper well-written (reviewer os8e, h9zL). Below, we address some commonly raised concerns:

**[Technical Contribution]** We recognize that there are differing opinions on the novelty of our proposed method. We would like to emphasize that our work proposes a *novel OOD detection framework* based on off-the-shelf generative models. Our framework is intentionally designed to be modular, keeping the object detector and the off-the-shelf generative model intact, while investigating how to combine them effectively for performing OOD detection in a zero-shot manner. By doing so, the framework can easily generalize to future object detectors and newly released generative models with minimal modification effort. We believe that prioritizing *simplicity* and *generalizability* enhances the impact and applicability of our work, rather than diminishing its contribution.

**[Runtime]** We report RONIN’s runtime in seconds per object. Additionally, we note that it is often possible to construct shorter diffusion schedules that maintain comparable performance. To illustrate the speed-performance tradeoff, we also present RONIN’s results using a 2-step inpainting schedule.

**Number of Seconds Per Object Processed**

|ID-OOD setting|RONIN with 20 steps (original schedule)|RONIN with 2 steps (improved schedule)|
|-|-|-|
|VOC-COCO|0.35|0.034|
|VOC-OpenImages|0.42|0.041|
|BDD-COCO|0.19|0.018|
|BDD-OpenImages|0.18|0.017|

**OOD Detection Performance**

|                  | RONIN (20 steps)          | RONIN (2 steps)          | 
|-------------------|----------------|----------------|
|                   | FPR@95 / AuROC | FPR@95 / AuROC |
| VOC - COCO       | 27.42 / 91.09  | 37.11 / 87.54  | 
| VOCO - OpenImages | 18.04 / 93.34  | 28.48 / 90.52  | 
| BDD - COCO        | 30.16 / 92.77  | 28.57 / 92.98  | 
| BDD - OpenImages  | 30.00 / 91.60  | 31.11 / 91.37  | 


**[Comparison with More Recent Methods]** We thank the reviewers for suggesting comparisons with more recent baselines. Below, we report a comparison with CLIPN [1]. Results show that RONIN consistently outperforms CLIPN.

|                   | CLIPN          | RONIN          |
|-------------------|----------------|----------------|
|                   | FPR@95 / AuROC | FPR@95 / AuROC |
| VOC - COCO        | 43.09 / 85.45  | 27.42 / 91.09  |
| VOCO - OpenImages | 41.74 / 89.31  | 18.04 / 93.34  |
| BDD - COCO        | 28.89 / 92.14  | 30.16 / 92.77  |
| BDD - OpenImages  | 44.76 / 85.78  | 30.00 / 91.60  |

Thanks again for all the insightful comments and suggestions. Please do not hesitate to let us know if there are additional concerns. We will do our best to address them in the remaining rebuttal period.

[1] Wang et al. "Clipn for zero-shot ood detection: Teaching clip to say no." ICCV 2023.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732727093125

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response on a revised manuscript

comment:
We sincerely thank all reviewers for their thoughtful comments and feedback. Based on their suggestions, we have revised our manuscript to address most, if not all, of their major concerns. The revised sections are highlighted in blue for clarity. Additionally, we will continue to address any remaining concerns promptly during the rebuttal period. If there are any remaining concerns that are not addressed, please let us know.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a novel zero-shot out-of-distribution detection method for object detectors based on inpainting object bounding boxes with stable diffusion. If class-conditioned inpainting deviates more from the original this is taken as an indicator of the wrong class being used for conditioning. 
The reviewers report several strengths: interesting idea to use inpainting for OOD, improvements over baseline, 
Weaknesses include: limited technical novelty, missing comparison to recent methods, sensitivity to hyperparameters, and inference time.