ReFOCUS: Recurrent False Object Correction Using guidance Strategies in Object Detection

Authors: Quentin Bourbon, Timothée Blondiaux, Joël Tang, Laurent Lam, Seif Edinne LAATIRI
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Object Detection, False Positive, Computer Vision, Recurrent Errors, Correction

Abstract:
This work addresses the issue of recurrent false positive classification in object detection. We consider two experimental setups imitating real-world scenarios that lead to such errors: i) erroneous annotations, ii) non-objects that resemble actual objects. We show that resulting models can be corrected efficiently using a two-step protocol that leverages false positive annotations. For the first step, we present and compare two correction approaches that guide false positives toward true negatives, in either the latent or the logit space. The second step then consists in standard continuous fine-tuning on correct annotations. The latent guidance framework relies on a decoder that maps the bounding box of a given false positive to its target true negative embedding. The decoder is trained as part of an autoencoder, where appropriate true negative samples are generated by a learnable Gaussian mixture model in the latent space. By leveraging the properties of the Wasserstein distance, the mixture model is optimized through standard backpropagation. In both experimental setups, the two correction methods significantly outperform standard continuous fine-tuning on correct annotations and demonstrate competitive performance when compared to models retrained from scratch on correct annotations. In particular, in the second experimental setup, the latent guidance framework consistently outperforms these models, effectively enhancing detection performance at the cost of supplementary false positive annotations. Additionally, the proposed techniques prove effective in a few-shot learning context.

Decision: reject
Reviewer scores: 3, 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1730368103324

thread_type:
official_review

<thread_object>
creator: reviewer_ypbg

<content>
summary:
This paper aims to correct false positives in object detection. Starting from erroneous annotations and non-bojects similar to the true samples, the model is corrected in the latent and logit spaces.

strengths:
1. The paper provides a rich theoretical basis and source of motivation.
2. The paper verifies the effectiveness of the proposed method in both general object detection and few-shot object detection scenarios.

weaknesses:
1. The paper mentions corrections in two spaces in both the abstract and contributions, but lacks a description of logit.
2. The paper mentions that ‘we assume that we have access to a corrective dataset DC’. It seems that the paper's method relies on clear FP data with the same data domain as the initial dataset to guide GMM generation and model fine-tuning, but such data is often not easy to obtain. We often cannot access the data after training a network, not to mention the FP information. How to solve this?
3. The paper proposes multiple loss functions in the method stage. It is recommended that the paper mark the different stages and training parameters of each loss in the Figure to facilitate readers' understanding. For example, in the final loss L_decoder stage, the author's network contains two MLP_bbox modules. Which one does ybbox target?
4. In Table 2, it seems that the author's method does not improve significantly compared to directly using f_Ture. Did the author compare the extra cost of this method compared to f_True? For example, a time comparison for such a two-stage process.

questions:
Before the Definition 1, the paper set the TN ‘only its position remains to be defined’, but after Definition 1, the paper said the TN ‘shares the same bounding box as the FP.’ How does this information align?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730533593886

thread_type:
official_review

<thread_object>
creator: reviewer_mrkq

<content>
summary:
This work addresses the issue of recurrent false positive (FP) classification in object detection. The paper proposes two innovative correction frameworks that guide FPs toward TNs in either the latent space or the logit space. The latent guidance framework leverages an autoencoder where a learnable Gaussian mixture model generates the embeddings of appropriate TNs, and a straightforward decoder retrieves the TN embedding given a bounding box. The paper utilizes the properties of the Wasserstein distance to train the Gaussian mixture model through standard backpropagation.

strengths:
1. The experiments show that the propsoed module improves the performance.
2. The idea of introducing other spaces other than logit space to solve classification errors is valuable.

weaknesses:
1. Figure 1 is the network diagram in DETR paper, without any information increment, so this content should not appear in the main text. In addition, although illustrations of motivation may not be necessary in the Introduction section, the description of the motivation section in this article should have an image which includes:
(a) the case visualization and explanation of the problem of false positives in classification errors. 
(b) the comparison with previous practices, as said in 'Motivations' part in Line 68. Tell the reader how the proposed method addresses the problem compared to previous approaches.
(c) the visualization of the noisy dataset D_{Noisy}, and well-annotated dataset D_{True}. And how they are used in the method.

2. Line 062 said the method 'can be generalized across different datasets and detection frameworks.' However, there were no experiments conducted on additional datasets and detectors in the experimental section and supplementary materials. In addition, both the data volume and the number of categories of VOC07 are not representative enough. Can it be validated with a larger dataset  (e.g., COCO, OpenImages)  and extra detectors to  better demonstrate the method's generalizability? Also, could you provide the computational requirements for testing on larger datasets if that is limited in larger dataset experments.

3.  The writing and paper organized (like the structure of method section) should be improved. For example:
(a). Sec 3.3.1 has 'Definition 1' (Line 221), then what? There are no  'Definition 2' in the following part.
(b). The relationship between sections in method. You can provide  an overview of  'LoGF' and 'LaGF' in 'CORE CONCEPT' part and tell us the structure of the following sub-sections.
......
and, there are too many colloquial words in the article, such as "we" appearing 107 times.

4. More visual displays are necessary. You don not need to show the results here, but rather make the before-and-after comparisons of specific false positive cases, or visualizations of how the latent space changes with the proposed methods.  In addition, some qualitative experiments on false positives can be added, such as analyzing whether the proportion of false positives decreases after adding methods from multiple dimensions, and so on.

questions:
See weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650191543

thread_type:
official_review

<thread_object>
creator: reviewer_mm1g

<content>
summary:
The work focuses on guiding the false positive object predictions to true negative predictions. The paper shows that models with FP predictions can be efficiently corrected using FP annotations. The paper proposed two correction approaches that guide false positives toward true negatives, i.e., in latent space (LoGF) and in logit space (LaGF). Both two methods required a corrective dataset where all recurrent FPs are additionally annotated. LoGF only modifies the classification logits of FP. LaGF uses an autoencoder architecture to change FP latents (sampled in a trained GMM) to TN latents (decoded by a trained decoder). After the training of the decoder, the original detection model is guided with newly annotated FP labels and  the decoder to bridge the gap between FP and TN in latent space, therefore, avoiding the FP predictions.

strengths:
The paper explores a new area to show that models with FP predictions can be efficiently corrected using FP annotations with two proposed methods, LoGF and LaGF.

weaknesses:
1. Unfair comparison. While these two proposed guidance frameworks both rely on a corrective dataset where all recurrent FPs are additionally annotated, a fair comparison should be a noisy model fine-tuned on the combination of corrective dataset D_c and correct dataset D_{True} instead of fined-tuned only on correct dataset D_{True}.
2. The experiments only explore the situation when only one pair of classes are misannotated, however, in the real world, multiple misannotated pairs are more common.
3. Some mistakes in writing. In Section 4.1, the authors use PASCAL VOC 2007, while in Section 5.1, the authors say to train the model on MS-COCO 2017. And no labels on the caption of tables to point out whether the results are done on the former or the latter dataset.

questions:
1. Please refer to the weakness.
2. Why contrastive learning is listed as one subsection of related works?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731052281030

thread_type:
official_review

<thread_object>
creator: reviewer_7k3h

<content>
summary:
In this paper, two strategies of improving object detectors are proposed to handle the issue caused by erroneous annotations and false positives. It is achieved by guiding the false positives toward true negatives in the latent or the logit space. By further fine-tuning on correct annotations, the proposed methods improve the detection performance in most cases.

strengths:
1. Two strategies of improving object detectors are proposed.
2. Training details are provided to ensure its reproducibility.

weaknesses:
1. The writing is poor. The authors need to carefully revised the paper from aspects of logical structures and grammars.
2. In the first sentence of abstract, “recurrent false positive classification” is required to be further explained. In the first paragraph of introduction, the explanation of “recurrent errors” is also needed. Are these two concepts have the same meaning?
3. The motivation may be questionable. In introduction, the authors claim that “the model consistently detects an object that should not be identified, e.g. people on billboards as instances of real people.” However, the phenomenon that people on billboards are detected may be reasonable in some situations. Classifying these samples to be background by force could increase the risk of model oscillation or overfitting.
4. In introduction, the Motivation paragraph seems to be logically incorrect. It seems that D_{True} is used in all cases. Why not using the f_{True} directly? A detailed explanation of this paragraph is required. Furthermore, the proposed methods only slightly outperform f_{True}. In some cases, the performance is even dropped, which limits the contribution of this paper.
5. The experiments are not convincing. For PASCAL VOC dataset, there are 20 classes of objects. However, the authors only uses a small part of them. The mAP on all classes should be reported.

questions:
The motivation of this paper should be re-clarified. The complete experimental results should be reported. The overall contribution is limited. Furthermore, this paper needs a major revision by re-organizing and proofreading its paragraphs.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The goal of the paper is to improve object detectors using two approaches: LoGF (latent space) and LaGF (logit space). These methods guide false positives  toward true negatives by modifying classification logits or using an autoencoder to change false positives  latents to rue negatives latents. By fine-tuning on correct annotations, the proposed methods can efficiently correct models with false positives  predictions. Reviewers seem to agree that the latent space method is novel and like the theoretical aspects of the paper. However, all reviewers seem to have comments regarding the writing and presentation of the paper. Most are also not convinced about the experiments. Hence the paper is not ready in its current form for ICLR.