Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a method of correcting the missing label issue in the context of few-shot instance segmentation and few-shot object detection problems.
strengths_and_weaknesses: Strengths:

- The paper is very well focused on solving one specific issue
- reasonably clearly written
- empirical framework is solid
- SOTA results
- Section "Why DC works?" seems to provide solid quantitative evidence to support the main claim of the paper.

Weaknesses:

- Contribution is rather shallow and incremental, the paper almost reads as a workshop paper in some respects. It is hard to evaluate the significance of the work, because problem being solved is not well articulated (see continuation below)
- problem being solved is not well articulated
  - In Fig 1 it is unclear why one of the person instances is mislabeled
  - it is unclear if the label noise is due to the dataset deficiency or it is due to the incorrect sampling scheme used to train few-shot methods. Is it possible to rectify the sampling scheme for few shot to avoid label noise? How does it happen that the objects that are not supposed to used as supervision signals end up being used as such?
  - It is unclear why this problem is solved architecturally by introducing additional heads while it seems like it is really a problem with the data. I would much prefer if the problem could be solved in the source, while in this case it looks like the data problem remains and instead of solving it directly, a heuristic is added at the model level to account for it
- Missing qualitative examples showing when the proposed technique succeeds in solving the stated problem and when it fails
- Since the proposed technique is claimed to solve a high level issue, applying it to a few base methods and showing that it works seems very important. However, this paper only shows results based on Mask-DeFRCN. 
questions: - The notation in equations 11-13 is confusing: equation 11 is the derivative of $L_{cls}^{fg}$, equation 12 is the derivative of $L_{cls}^{fg}$ and equation 13 is the derivative of $L_{cls}^{bg}$, while on the LHS there is always $L_{cls}$
- How easy it is to extend other existing backbones with the proposed technique? What does it take in terms of actual code changes? Can you provide a clean code example?
limitations: N/A
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper addresses the missing label issue in few-shot object detection (FSOD) and few-shot instance segmentation (FSIS). Some foreground examples are not labeled in the few-shot scenario, which causes classification bias for the conventional classification head used in object detectors. This paper proposes a simple yet effective algorithm that treats the foreground and background proposals differently in the classification loss. The proposed method is evaluated on benchmark datasets and shows competitive performance against SOTA methods for both FSOD and FSIS.
strengths_and_weaknesses: Pros:

1. The proposed method is very simple yet effective. It decouples the loss function for foreground and background objects to remedy the effect on the classification head from the missing labels for potential foreground objects. The evaluations on benchmark datasets demonstrate its effectiveness in both FSOD and FSIS.
2. The authors show that the proposed loss can be incorporated into other FSOD methods to achieve better performance. It also has no negative impact on base classes per the evaluation results. It might become a plug-in for many FSOD methods.
3. The paper is generally well written and easy to follow.

Cons:

1. While I acknowledge the issue of missing labels in the FSOD, it is more like a bad design in the current FSOD evaluation protocol rather than a real-world problem. It is weird when you have a lot of images, but you only label a single class and ignore others, especially when you know it will have a negative impact. If you have limited resources (as in a few-shot setting), shouldn't you focus on labeling a small group of images to ensure every class is labeled so that the total # labels are roughly similar? I understand the protocol is designed to control the number of shots, but it just seems unrealistic. 
questions: 1. If the novel class training samples are provided in a way that every example is labeled in each image but only a small group of images are labeled (basically the strategy I mentioned in Cons), do you think the proposed method works better or worse? Also, will other FSOD/FSIS methods work better or worse in this case? Do you think using the proposed method with the current labeling strategy (i.e. randomly labeling a single class and ignoring others in one image) is still more beneficial as it sees more data?
limitations: The authors discussed the limitations
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposes a decoupled classifier to deal with positive and negative proposals separately when learning general few-shot object detection (FSOD) and few-shot instance segmentation (FSIS) networks. Specifically, the paper argues that for FSOD/FSIS problem, missing labels are a common issue. For an image, some classes could be ignored during annotation. The missing annotation problem would lead to a biased classifier, especially when learning with a few images. To solve this problem, the paper proposes to deal with positive proposals with normal cross-entropy loss and deal with negative proposals by only considering the positive class and background of the images.

The proposed method is implemented based on the architecture of DeFRCN [21] and evaluated on benchmark datasets VOC 15/5 and COCO 60/20. The method is compared with several SOTA methods, including DeFRCN, and appears to be better. 

Overall, the idea of the paper should be easily reproducible. The benchmark, evaluation, and comparison are solid. However, I think there are several major issues with the premise, and the proposed method could lead to more bias in certain situations. Details can be found in the  Strengths And Weaknesses section.
strengths_and_weaknesses: The main strengths of the paper are:

1. The paper is relatively easy to follow. The proposed method is described in detail and should be easily reproducible.
2. The comparisons and evaluation are adequate, and the results are positive on the benchmark.

The main weaknesses of the paper are:

1. My major concern with the paper is the premise. I think the paper proposes a solution to deal with the bias of existing benchmark datasets rather than the problem (FSOD/FSIS) themselves. In particular, in Fig 1(c), the paper mentions that for a one-shot image, 'dog' could be annotated and the 'person' could be ignored.  This situation is mainly due to how existing benchmarks (VOC/COCO) are proposed. Prior works [14, 26] proposed the benchmarks that consider an object instance (bounding boxes) as a 'shot'. In this case, the training images could just be sampled with one instance, even though there are multiple instances in the same image. I agree with the paper that this is an issue with these benchmarks and needs to be addressed. However, I do not think this is an inherent issue with FSOD/FSIS problems.  
2. Even if the problem is inherent with FSOD, the proposed solution could create more bias rather than solving the problem. With the same example in Fig 1(c). If it is a one-shot scenario for the class 'person', then the other person might not be annotated. In fact, this is a very common problem for object detection, even for large-scale datasets. There are plenty of examples of 'group' bounding boxes with a single example instance in the OpenImages dataset (https://storage.googleapis.com/openimages/web/index.html).  These missing annotations would be considered negative ('background') and the proposed method restricted the learning to between ['person', 'background'], and backpropagate wrong losses. I think in many situations, the proposed method can lead to more bias and confusion.
3. Some of the paper's claims are not well-grounded. For example, Line 94-95, the paper claims that " It is impractical from the perspective of many real-world applications because ones always expect that few shot model is capable of not only recognizing novel classes but also remembering base classes". However, for many few-shot learning papers (MAML [a] and Prototypical Network [b] for example), base-class performance is not a concern. Could the author motivate more on why we 'always expect the few-shot model to remember base classes‘？

[a] Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks, Finn et al, ICML 2017
[b] Prototypical Networks for Few-shot Learning,  Snell at al, NeurIPS 2017
questions: Please refer to the weakness section for questions.
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper formally proposes the missing label issue which naturally happens in few-shot scenarios including few-shot object detection (FSOD) and few-shot instance segmentation (FSIS). The missing label issue may result in biased classification (mistakenly recognizing novel class objects as background, i.e., missing detection) and thus reduce the generalization performance of many current FSOD/FSIS models.  From the missing label perspective, the paper presents a simple but effective method that decouples the standard classifier into two parallel heads which are capable of addressing clear positive samples and noisy negative samples, respectively. Using the proposed decoupling classifier, the model can effectively learn novel classes because the effects of noisy negative samples are well mitigated. The effectiveness of the proposed method is evaluated on standard few-shot benchmarks PASCAL VOC and MS COCO for both FSOD/FSIS and generalized FSOD/FSIS tasks.
strengths_and_weaknesses: Strengths:
+ The paper observes the fact that the missing label issue naturally exists in few-shot scenarios such as FSOD and FSIS.  Furthermore, the authors analyze the missing label issue may result in biased classification and reduce the generalization performance of FSOD and FSIS. This new perspective might be inspiring to others in the field. 

+ In order to mitigate the effects of noisy negative samples and reduce the biased classification in FSOD and FSIS, the paper proposes a novel classifier decoupling idea that is quite interesting and strongly motivated. It can individually address clear positive samples and noisy negative samples and thus mitigate the biased classification issue. The proposed method is really simple and easy to understand and follow.

+ The performance of the proposed method outperforms its baseline and state-of-the-art by a large margin on PASCAL VOC and MS-COCO benchmarks for both FSOD/FSIS and generalized FSOD/FSIS tasks. Meanwhile, the proposed method doesn’t introduce any additional computation costs and hyper-parameters because it only simply decouples the standard classifier into two parallel classifiers. Therefore, it seems like a practical algorithm that can be put immediately into practice.

+ The proposed method is well analyzed and interpreted from the gradient optimization perspective. On the other hand, the authors evaluate the performance of the learned classifier using the Recall metric for all ground-truth foreground objects of all testing images and demonstrate that the proposed decoupling classifier indeed mitigates the biased classification. 

+ Ablation studies cover all the crucial components of the proposed approach. The experiment results are quite convincing.

+ The manuscript is well written and the organization is very clarified.

Weaknesses:
- Relationship with some existing works~(e.g., DeFRCN). The missing label issue also, in fact, exists between base and novel classes. For example, the novel class objects may potentially present in base images at the base learning stage but these potentially novel objects are viewed as background and it is opposite to the base learning stage at the novel fine-tuning stage. Therefore, the missing label issue also leads to foreground-background confusion between base and novel learning stages because of this two-stage fine-tuning mechanism. I understand that this paper focuses on the missing label issue at the novel fine-tuning stage. However, it is more helpful for the readers to understand the proposed method if these two types of missing label issues can be comprehensively analyzed.

- It is unclear how to use the decoupling classifier at the inference stage because the image-level few-shot label ($m_i$ in Eq. 7) is agnostic for any testing images. I guess that the standard classifier is used at the inference stage and the proposed decoupling classifier only is employed during training time. The authors should clarify it.

questions: - This missing label issue should be comprehensively analyzed from a broad perspective, e.g., between base and novel classes stages, the novel fine-tuning stage itself.

- The inference pipeline (using the standard classifier or the decoupling classifier) should be more clarified.

limitations: Yes. The authors have discussed the limitations of the proposed methods in Sec.4 (Experiments) and Sec.5 (Conclusion). As stated in the paper, It may not be suitable when the missing label rate is small. However, the proposed method is still comparable to its counterpart even if the missing label rate is zero, which indicates the robustness of the proposed method. Considering that it is unavoidable to meet the missing label issue in few-shot FSOD and FSIS, especially under multiple categories scenarios and so it is ok for me.
review_title: null

Meta-review: metareview: **Summary**: This paper aims to address the missing label issue in few-shot object detection (FSOD) and instance segmentation (FSIS). In these tasks, some foreground examples (more specifically, classes) are not labeled in the training images, which causes classification bias for the conventional classification head. This paper proposes a simple yet effective algorithm that treats the foreground and background proposals differently in the classification loss. The proposed method is evaluated on benchmark datasets and shows competitive performance against SOTA methods for both FSOD and FSIS.

**Strength**: The paper is well-written. The observation is interesting and inspiring. The proposed method is novel, simple, effective, well-motivated, and compatible. The empirical study is solid and achieves SOTA results. The proposed method introduces no additional computation costs and hyper-parameters and is thus practical.

**Weakness**: The problem being solved is not well articulated. The inherent problem may come from poor task/protocol design. The paper might propose a solution to deal with the bias of existing benchmark datasets rather than the problem (FSOD/FSIS) themselves. Some of the paper's claims are not well-grounded. This missing label issue should be comprehensively analyzed from a broad perspective.

**Recommendation**: The paper received mixed review opinions from four reviewers. On the one hand, reviewers found the observation interesting and the proposed method simple, effective, and well-motivated. On the other hand, reviewers also have concerns that the missing label issue may not be the true problem of FSOD/FSIS, but a side effect of the poorly designed benchmark. Indeed, this was the main concern of the two reviewers (mATA and 1weo) who gave 3 and 4. 

After the rebuttal, two reviewers are satisfied with the authors' responses to their concerns and raised their scores to 7. However, the above-mentioned two reviewers did not participate in the discussion.

The AC carefully read the paper and thought that the missing label issue can be a natural/practical issue in FSOD/FSIS. Specifically, when one wants to further detect an object class that is not in the base classes, one natural way to collect data is to find images that contain that object class and just annotate it (and ignore annotating other classes). Given this, the strengths of the paper, and the acceptance suggested by two reviewers, the AC suggests “acceptance.” 

With that being said, the AC strongly suggests that the authors take the comments by reviewers mATA and 1weo seriously. Those questions, though might be beyond the scope, are quite valuable. For example, reviewer 1weo asks 1) if it is beneficial to collect more images (but each with partial labels) than fewer images (but each with full labels); 2) if other FSOD/FSIS methods work better or worse when only a few fully annotated images are provided. The AC respectfully thinks that the authors misunderstood the questions and unrelated responses. 

Also, the authors' response to reviewer mATA can be improved. The authors said, “Second, the community accepts the current FSOD/FSIS benchmarks because they are more challenging due to incomplete or partial annotations (i.e., missing labels).” However, more *challenging* does not mean that the problem setting is appropriate/proper/practical. Similarly, *commonly used* benchmarks do not mean that they don't have problems in the setting. In response to reviewer mATA’s question, the authors could have focused more on why this setting is valid, practical, etc.

Overall, the AC sees the strength and value of the paper. To make the paper stronger and more impactful, the AC has the following suggestions. First, the AC suggests that the authors incorporate all the reviewers' comments and the authors' rebuttal into their final version. Second, the authors should add content that they promise to the reviewers. Third, the authors should add a paragraph to clarify the missing label issue. Fourth, the authors should further discuss the relationship to semi-supervised learning (e.g., the pseudo-label methods), which could potentially handle missing label issues. 

confidence: Less certain
Decision: Accept