Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Durning ﬁne-tuning of few-shot object detection, a novel class may implicitly leverage the 8 knowledge of multiple base classes to construct its feature space, which induces a scattered feature space, hence violating the inter-class separability. To overcome these obstacles, we propose a two-step ﬁne-tuning framework, Few-shot object detection via Association and DIscrimination (FADI), which builds up a discriminative feature space for each novel class with two integral steps. 
main_review: I am in general positive towards this paper, as the method has convincing motivation, and is presented very clearly. 

I still have a few comments: 

Authors claim that existing SOTA few-shot detection works all employ a finetuning strategy, this is not appropriate. There are still several works focusing on the meta-learning mode, for instance, RepMet [13] and NP-RepMet [NIPS2020]. 

For the class similarity measure, authors choose WordNet over over visual similarity.  From my viewpoint, visual similarity also has advantages that WordNet does not. For instance, an airplane and a bird can only be similar visually.  

For the feature distribution alignment, what's the motivation to implement it this way ? Have you tried some other ways, for instance, gradient reversal layer ?  

The method performs SOTA on low-shots in most scenarios, yet not in the second split of VOC. Any discussions and observations on this ? 
Also, have authors tried to integrate their methods into FCSE to see if they could further achieve SOTA on rather higher shots ? 
 
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: 
The paper proposes a method for few shot object detection. The key idea is to leverage the similarity of a novel object class with an existing base class, and then train a local classifier for better discrimination. The class similarity is based on semantic word similarity between the classes’ textual names and the new class to base class mapping is an injective mapping. Once the assignment is calculated the feature network is trained so that the features of the new class align with its corresponding base class. Finally further discriminator layers are used which are then trained to classify base and new classes distinctly. Empirical results are shown on PASCAL VOC 2007 2012 and COCO datasets, where the proposed method outperforms existing methods especially in very low shot settings.

main_review: The main motivation seems to be that if you fine-tune with the less amount of data, then the local optima of the resulting optimization might not be good. So the idea is to constrain the optimization region with priors. The first being that the feature network should be highly biased to keep similar classes in the same locality in both base and new classes. Second one being that the classifiers for the similar base and novel classes should be in some sense trained locally.

The colors used in fig4 could be better, I had to look carefully to see that book and cow are two different colors and not different weights of different colors (first reflex is to associate light color with small weight, on which I had to remind myself that the association is binary, right?). Use contrasting colors. And perhaps for the non matched classes have highlights or underlines or some other decoration to make it easy to perceive.

Also in fig4 score confusion matrix is shown, but what exactly is score confusion is not defined? It is also not very clear why the left column on shows base classes in the x axis which the right col has both base and new ones.

In table 5 association comparison with visual features derived similarity should also be compared with the semantic text similarity. It would be interesting to see how much worse is that.

The tab6 argument seems weak as in one shot situation, we would expect the variance to be very high, and the numbers to flicker a lot depending on the splitting. With even a slight increase in number of examples the visual and textual similarities seem to be similar.

The AP is reported only on novel classes (nAP). As experienced with zero and few shot classification works, there is usually a tradeoff between novel class only performance and whole set performance. I understand that practitioners may be only interested in new class performances, but academically we should be aware of how the performances are suffering for old classes, and all classes taken together. Since those metrics should be easy to compute with the new model, I would strongly encourage them to be reported (even if only for the current method, as the older methods might not have reported them. I see that TFA has reported that on base and novel separately). Since the method here is advocates strong local learning, it might be alleviating confusion for new classes, but might be increasing the same for old classes. A discussion would also be appreciated.

limitations_and_societal_impact: n/a
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper proposes a novel approach for few-shot object detection by first assigning a novel (few-shot) class to a base category (Association) and then disentangling the classification of base and novel classes (Discrimination). After the first step of base training, the proposed approach assigns each novel class to a base class using the WordNet hierarchy. Then, the method aligns the feature distribution of the novel class to the associated base class. Finally, the method proposes to disentangle the classification branches of base and novel classes to remove confusion between base and novel classes. 
main_review: The proposed approach is highly intuitive and has been well described by the authors. I believe that this idea is novel and is a strong contribution to the field. The authors have also demonstrated the effectiveness of the proposed approach through strong results on the PASCAL VOC and MSCOCO datasets in the few-shot setting. 

I have a few questions/suggestion about the paper which the authors should address in their response. 

1. In Table 1, the authors show good improvements for low-shot learning in Split 1 and Split 3. Why is the performance not so good for 5 and 10 shots for Split 1? 5-shot seems to be particularly low compared to FSCE. Why is this the case? Was the performance of FSCE somehow artificially too high in this case or is the performance of FADI artificially too low? Further, why is there no similar improvement for Split 2? How is Split 2 different from Splits 1 and 2?

2. For Table 5, do the same observations hold for larger datasets like MSCOCO? Do these observation change with change in dataset?
limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative social impact of their work but only in the supplementary material and not in the main text of the paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a two-step fine-tuning framework for few-shot object detection, where the first stage is an association step, associating each novel class with a base class according to their semantic similarity, and the second stage is a discrimination step, which ensure the separability between the novel classes and associated base classes. Experiments show the effectiveness of the proposed method. 
main_review: The paper proposes a new framework for few-shot object detection and achieves good results on some indicators. Generally, the paper is well organized and written.

However, there are obvious limitations with this paper: 
1. Feature distribution alignment module is interesting and novel. However, the proposed set-specialized margin loss has limited innovation. The loss is very similar to cos similarity, which is also used in TFA. Each line of W in Eq.5 can be regraded as a prototype of a category. The main innovation of the proposed loss function lies in that the weights of the novel classes are increased by hyperparameter $\beta$, and the weight of the background class is reduced by hyperparameter $\gamma$. It's more like a technical trick. In line 198-201, the authors explained the motivation of reweighing. As can be seen from Appendix C, when $\alpha/\beta $ increases, the AP50 improves little (only from 49.6 to 50.3). Similarly, for the adjustment of gamma, the effect on the result is relatively small. Therefore, the validity of this technique is questionable. Moreover, the authors did not do experiment for $\gamma$ = 1 in Table 4 (b) of Appendix C.

2. In experiments, comparisons with the latest SOTAs are missed. Following are 
some papers in CVPR2020 have not been compared:
[1] Fan, Qi, et al. "Few-shot object detection with attention-RPN and multi-relation detector." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
[2] Yang, Yukuan, et al. "Restoring Negative Information in Few-Shot Object Detection." Advances in Neural Information Processing Systems 33 (2020): 3521-3532.
[3] Yang, Ze, et al. "Context-transformer: tackling object confusion for few-shot detection." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 07. 2020.

In addition, comparing the papers in CVPR 2021, the performance of the proposed method is not promising:
[4] Zhang, Weilin, and Yu-Xiong Wang. "Hallucination Improves Few-Shot Object Detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
[5] Li, Yiting, et al. "Few-Shot Object Detection via Classification Refinement and Distractor Retreatment." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
[6] Fan, Zhibo, et al. "Generalized Few-Shot Object Detection without Forgetting." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
[7] Li, Aoxue, and Zhenguo Li. "Transformation Invariant Few-Shot Object Detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
[8] Zhang, Lu, et al. "Accurate Few-Shot Object Detection With Support-Query Mutual Guidance and Hybrid Loss." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
[9] Hu, Hanzhe, et al. "Dense Relation Distillation with Context-aware Aggregation for Few-Shot Object Detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
[10] Li, Bohao, et al. "Beyond Max-Margin: Class Margin Equilibrium for Few-shot Object Detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

3. Line 181:  Where is $\widetilde{W}^B_{asso}$ from? As I understand, in the association step, we can obtain $\widetilde{W}^N_{asso}$, which is the intermediate structure that follows $\phi(;\widetilde{W}^B_{pre})$ (feature extractor). $\widetilde{W}^N_{asso}$ is used to perform feature distribution alignment for novel classes, but for base classes, it doesn’t need to perform feature distribution alignment. So, what does $\widetilde{W}^B_{asso}$ mean? In line 181-182, it’s said that it comes from the association step, but I cannot find the variable in the association step.

limitations_and_societal_impact: 1. If the semantic similarity between the novel class and all base class is small, there may be some problems with the association process. It is possible to choose the most similar one among the many dissimilar base classes. However, the selected base class may be not similar to the novel class. On the other hand, it is possible that the similarity between a novel class and multiple base classes is very high. In this case, there may be some problems when aligning it to one of the base classes.

2. In the association step, there may be a visual semantic gap. That is, some classes are semantically similar, but not visually similar. This problem is also common in zero-shot learning and zero-shot object detection [1]. However, the authors didn’t consider this issue.

[1] C. Yan, Q. Zheng, X. Chang, M. Luo, C. -H. Yeh and A. G. Hauptman, "Semantics-Preserving Graph Propagation for Zero-Shot Object Detection," in IEEE Transactions on Image Processing, vol. 29, pp. 8163-8176, 2020, doi: 10.1109/TIP.2020.3011807.

3. The experimental setting of this paper is different from the previous FSOD works. This work introduces the semantic information of the category, which is equivalent to having additional information input. There are some works in few-shot learning [2], and the introduction of semantic information can improve performance by a large margin. However, the performance improvement brought by semantic information in this paper is limited (in Tab.3, less than 1 point in 1/3-shot)，this makes the effectiveness of the method questionable. 

[2] Chen Xing, Negar Rostamzadeh, Boris N Oreshkin and Pedro O Pinheiro, "Adaptive cross-modal few-shot learning", Advances in Neural Information Processing Systems, pp. 4848-4858, 2019.
review_title: null

Meta-review: The paper proposes an alternative to fine-tuning for few-shot object detection, by instead associating each new class to an existing base class and then training with a max margin loss between this new class from the base class. The idea is simple, its implementation probably still leaves some performance on the table (e.g. visual similarity vs. WordNet similarity). The experimental validation is strong, in particular in term of AP on new classes on standard benchmarks.

The reviewers agree that the paper is mostly clear and convincing. The authors addressed some of the limitations of the paper in the author response, in particular in term of comparisons with the literature. I recommend acceptance as a poster.
Decision: Accept (Poster)