Few-Shot Object Detection via Association and DIscrimination
Authors: Yuhang Cao
Jiaqi Wang
Ying Jin
Tong Wu
Kai Chen
Ziwei Liu
Dahua Lin
Venue: NeurIPS 2021 Poster (2021)
Keywords: Few-Shot Object Detection
Association
Discrimination

Abstract: Object detection has achieved substantial progress in the last decade. However, detecting novel classes with only few samples remains challenging, since deep learning under low data regime usually leads to a degraded feature space. Existing works employ a holistic fine-tuning paradigm to tackle this problem, where the model is first pre-trained on all base classes with abundant samples, and then it is used to carve the novel class feature space. Nonetheless, this paradigm is still imperfect. Durning fine-tuning, a novel class may implicitly leverage the knowledge of multiple base classes to construct its feature space, which induces a scattered feature space, hence violating the inter-class separability. To overcome these obstacles, we propose a two-step fine-tuning framework, Few-shot object detection via Association and DIscrimination (FADI), which builds up a discriminative feature space for each novel class with two integral steps. 1) In the association step, in contrast to implicitly leveraging multiple base classes, we construct a compact novel class feature space via explicitly imitating a specific base class feature space. Specifically, we associate each novel class with a base class according to their semantic similarity. After that, the feature space of a novel class can readily imitate the well-trained feature space of the associated base class. 2) In the discrimination step, to ensure the separability between the novel classes and associated base classes, we disentangle the classification branches for base and novel classes. To further enlarge the inter-class separability between all classes, a set-specialized margin loss is imposed. Extensive experiments on standard Pascal VOC and MS-COCO datasets demonstrate that FADI achieves new state-of-the-art performance, significantly improving the baseline in any shot/split by +18.7. Notably, the advantage of FADI is most announced on extremely few-shot scenarios (e.g. 1- and 3- shot).

Few-Shot Object Detection via Association and DIscrimination
Yuhang Cao
Jiaqi Wang's 1.20 1.2
Ying Jin'
Tong Wu'
Kai 2hen Ziwei Liu
Dahua Lin
CUUK-S Seik--SenTTTmee Joint Lab, The Chinese University of Hong Kong
Shanghai AI Laboratory SSense Research
Ls-Lab, Nanyang Technological University
{cy020, , xj017, wt020, dhlin]@ie cuhk. edu. tkk
chenkai@sensetinn com ziwei. ziwei.line liu@ntu. edu.
Abstract
Object detection has achieved substantial progress in the last decade. However, detecting novel classes with only few samples remains challenging, since deep learning under low data regime usually leads to a degraded feature space. Existing works employ holistic fine-tuning paradigm tackle this problem, where the model is first pre-trained on all base classes with abundant samples, and then it is used carve the novel class feature space. Nonetheless, this paradigm is still imperfect. Durning fine-tuning, novel class may implicitly leverage the knowledge of multiple base classes construct its feature space, which induces a scattered feature space, hence violating the inter-class separability. To over- come these obstacles, we propose a two-step fine-tuning framework, Few-shot object detection via Association and DIscrimination (FADI), which builds up a discriminative feature space for each novel class with two integral steps. 1) the association step, in contrast to implicitly leveraging multiple base classes, we construct a compact novel class feature space via explicitly imitating a specific base class feature space. Specifically, we associate each novel class with base class according to their semantic similarity. After that, the feature space novel class can readily imitate the well-trained feature space of the associated hase class. 2) In the discrimination step, ensure the separability between the novel classes and associated base classes, disentangle the classification branches for base and novel classes. To further enlarge the inter-class separability between all classes, a set-specialized margin loss is imposed. Extensive experiments standard Pascal VOC and MS-COCO datasets demonstrate that FADI achieves new state-of-the performance, significantly improving the baseline in any shot/split by +18.7. 7. No- tably, the advantage of FÀDI is most announced on extremely few-shot scenarios (e.g. 1- and 3- shot). Code is available https: / thub. httpb..coohhheeeeeeeeee com/yHcao6/FADI
Introduction
Deep learning has achieved impressive performance on object detection [21, 11, 1] in recent years. However, their strong performance heavily relies on large amount of labeled training data, which limits the scalability and generalizability of the model the data scarcity scenarios. In contrast, human visual systems can easily generalize to novel classes with only few supervisions. Therefore, great interests have been invoked to explore few-shot object detection (FSOD), which aims at training network from limited annotations of novel classes with the aid of sufficient data of base classes.
"Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

CiP
Sheap
Shorp
Hore
Motur
Mater
Tike
likk
Bike
(a) TFA
(b) Association
(c) Discrimination
Figure I: Conceptually visualization of our FADI. (a) The conventional fine-tuning paradigm, e.g., TTA [28]. learns good decision boundaries during the pre- pre-training stage to separate the decision space into several subspaces (rectangles) occupied by different base classes. multiple similar base classes ( ('sheep' and 'horse') ') to construct the feature space of itself, which induces a scattered intra-class structure (the feature space of 'cow' across two base classes, 'sheep' and "horse'). ) FADI divides the fine-tuning stage into two steps. (b) In the association step, to construct compact intra-class structure, associate each novel class with well-learned base class based their semantic similarity ('cow' similar to 'sheep", 'motor' similar to 'bike"). The novel class readily learns to align intra-class distribution to the associated base class. (c) In the discrimination step. to ensure the inter-class separability between novel classes and associated base classes, we disentangle the classification branches for base and novel classes. A set-specialized margin loss is further imposed to enlarge the inter-class separability between all classes.
the fine-tuning stage, a novel class ('cow") may exploit
Various methods have since been proposed to tackle the problem of FSOD, including meta-
learning [13, 35, 32], metric learning [14], and fine-tuning [28, 31, 23]. Among them, fine-tuning- based methods are one of the dominating paradigms for few-shot object detection. [28] introduces simple two-stage fine-tuning approach (TFA). MPSR [31] improves upon TFA [28] via alleviating the problem of scale variation. The recent state-of-the-art method FSCE [23] shows the classifier is more error-pre proo-ppone than the regressor, and introduces the contrastive-awee aware object proposal encodings to facilitate the classification of detected objects. All these works employ a holistic fine-tuning paradigm. where the model is first trained on all base classes with abundant samples, and then the pre- pre-trained model is fine-tuned on novel classes. Although it exhibits considerable performance advantage compared with the earlier meta-learning methods, this fine-tuning paradigm is still imperfect. To be specific, the current design of the fine-tuning stage directly extracts the feature representation of novel class from the network preitrained on base classes. Therefore, novel class may exploit the knowledge of multiple similar base classes to construct the feature space of itself. As a result, the feature space of novel class will have incompact intra-class structure that scatters across feature spaces of other classes, breaking the inter-class separability, hence leading to classification confusion, as shown in Figure la.
To overcome these obstacles, we propose two-step fine-tuning framework, Few-shot object detection via Association and DIscrimination (FADI), which constructs a discriminable feature space for each novel class with two integral steps, association and discrimination. Specifically, the association step, as shown in Figure 1b, to construct compact intra-class structure, we associate each novel class with well- trained base class based on their underlying semantic similarity. The novel class readily learns to align its feature space to the associated base class, thus naturally becomes separable from remaining classes. the discrimination step, as shown in Figure Ic to ensure the separability between the novel classes and associated base classes, we disentangle the classification branches for base and novel classes to reduce the ambiguity in the feature space induced by the association step. To further enlarge the inter-class separability between all classes, set-specialized margin loss is applied. To this end, the fine-tuning stage is divided into two dedicated steps, and together complement each other.
Extensive experimental results have validated the effectiveness of our approach. We gain significant performance improvements on the Pascal VOC [7] and COCO [18] benchmarks, especially on the extremely few-shot scenario. Specifically, without bells and whistles, FADI improves the TFA [28] baseline by significant margin in any split and shot with up to + +18.7 7 mAP, and push the envelope of the otfte-of-th performance by 2.5, 4.3, 2.8 and 5.6, 7.8, 1.6 for shot K = 1,2, 3 on novel split-1 and 3plit-3 of Pascal VOC dataset, respectively.

---

2 Related Work
Few-Shot Classification Few-Shot Classification aims to recognize novel instances with abundant base samples and few novel samples. Metric-based methods address the few-shot learning by learning to compare, different distance formulations [26, 22, 24] are adopted. Initialization-baan based methods [9, 15] learn good weight initialization to promote the adaption to unseen samples more effectively. Hallucination-base methods introduce hallucination techniques [10, 29] to alleviate the shortage of novel data. Recently, researchers find out that the simple pre-training and fine-tuning framework [4, 6] can compare favorably against other complex algorithms.
Few-Shot Object Detection As emerging task, FSOD is less explored than few-shot classifi- cation. Early works mainly explore the line of meta-learning [13, 35, 32, 14, 36, 37, 8), where a meta-learner is introduced to acquire class agnostic meta knowledge that can be transferred to novel classes. Later, [28] introduces a simple two-stage fine-tuning approach (TFA), which significantly outperforms the earlier meta-learning methods. Following this framework, MPSR [31] enriches object scales by generating multi-scale positive samples to alleviate the inherent scale bias. Recently, FSCE [23] shows in FSOD, the classifier is more prror-prone than the regressor and introduces the contrastive-awars object proposal encodings to facilitate the classification of detected objects. Similarly, FADI also aims to promote the discrimination capacity of the classifier. But unlike previous methods that directly learn the classifier by implicitly exploiting the base knowledge, motivated by the works of [34, 331, FADI explicitly associates each novel class with a semantically similar base class to learn a compact intra-class distribution.
Margin Loss Loss function plays an important role in the field of recognition tasks. To enhance the discrimination power of traditional softmax loss, different kinds of margin loss are proposed. SphereFace [19] introduces a multiplicative margin constrain in a hypersphere manifold. However, the non-monotonicity of cosine function makes it difficult for stable optimization, CosFace [27] then proposed to further normalize the feature embedding and impose an additive margin in the cosine space. ArcFace [5] moves the additive cosine margin into the angular space to obtain better discrimination power and more stable training. However, we find these margin losses are not directly applicable under data-scarce settings they equally treat different kinds of samples but ignore the inherent bias of the classifier towards base classes. Hence we propose a set-specialized margin loss that takes the kind of samples into consideration which yields significantly better performance.
3 Our Approach
In this section, we first review the preliminaries of few-shot object detection setting and the conven- tional two-stage fine-tuni mnn-ttning framework. Then we introduce our method that tackles few-shot object detection via association and discrimination (FADI).
3.1 Preliminaries
In few-shot detection, the training set is composed of base set D= =
with abundant data
of classes CB and novel set D {=:" v with few-shot data of classes CN where x, and ya
indicate training samples and labels, respectively. The number of objects for each class in CN is K for K-shot detection. The model is expected to detect objects in the test set with classes in C U C
Fine-tuning- based methods are the current one of the leading paradigms for few-shot object detection, which successfully adopt a simple two-stage training pipeline to leverage the knowledge of base classes. TFA [28] is widely adopted baseline of fine-tuning-bast
few-shot detectors. In the base training stage, the model is trained on base classes with sufficient data to obtain a robust feature representation. In the novel fine-tuning stage, the pre-trained model on base classes is then fine-tuned on a balanced few-shot set which comprises both base and novel classes (CB U Cx). Aiming at preventing fiet-fitiig during fine-tuning, only the box predictor, i.e., classifier and regressor, updated to fit the sho-shot set. While the feature extractor, i.e., other structures of the network, are frozen [28] to preserve the pre-trained knowledge on the abundant base classes.
Although the current design of fine-tuning stage brings considerable gains on few-shot detection, we observe that it may induce scattered feature space on novel class, which violates the inter-class
3

---

Step1: Association
Pre-trained
Base Class
Novel Class
aeroplane
Pseudo Label
Frozen
train
bus
Ge
Ro! Features
FC
FC2
Base Classifier
train-bus
bicycle
Frozen
g W
f(: W
Pseudo Label
horse
cow
Step2: Discrimination
Rorse-cco
Rand Init
Stage1
g(+ W
origin
Regressor
aeroplane
train
FC
Base Classifier
bicycle
Margin Loss
f(: Waa)
Features
Roi
FC₁
Disentangle
horse
FC2
Novel Classifier
Frozen
Cow
9 ( W asso
f(: W.:)
Figure 2: Method overview. There two steps in FADI: association and discrimination. To construct compact intra-class structure, the association step aligns the feature distribution of each novel class with well-learned base class based their semantic similarity. ensure inter-class separability, the discrimination step disentangles classification branches for base and novel classes and imposes a set-specialized margin loss.
separability and leads to confusion of classification. Towards this drawback, we proposes few-shot object detection via association and discrimination (FADI), which divides the fine-tuning stage of TFA into two-step association and discrimination pipelines. In the association step (Sec. 3.2), to construct a compact intra-class distribution, we associate each novel class with base class based on their underlying semantic similarity. The feature representation of the associated base class is explicitly learned by the novel class. In the discrimination step (Sec. 3.3), to ensure the inter-class separability, disentangle the base and novel branches and impose set-specialized margin loss to train a more discriminative classifier for each class.
3.2
Association Step
In the base training stage, the base model is trained on the abundant base data DB and its classifier learns a good decision boundary (see Figure 1) to separate the whole decision space into several subspaces that are occupied by different base classes. Therefore, if novel class can align the feature distribution of a base class, will fall into the intra-class distribution of the associated base class, and be naturally separable from the other base classes, And if two novel classes are assigned to different base classes, they will also become separable from each other.
To achieve this goal, we introduce new concept named association, which pairs each novel class to similar base class by semantic similarity. After then, the feature distribution the novel class is aligned with the associated base class via feature distribution alignment.
Similarity Measurement In order to ease the difficulty of feature distribution alignment, given a novel class C.N and a set of base classes C we want to associate C² to the most similar base class in C An intuitive way is to rely on visual similarity between feature embeddings. However, the embedding is not representative for novel classes under data-scarce scenarios. Thus, adopt WordNet [20] as an auxiliary to describe the semantic similarity between classes. WordNet is an English vocabulary graph, where nodes represent lemmas synsets and they are linked according their relations. It incorporates rich lexical knowledge which benefits the association. Lin Similarity [16] is used calculate the class-to-class similarity upon WordNet which is given by:
IC(LCS(CN (B))
sim(C, ,,,)
IC(CN) + +CC(C))
where LCS denotes the lowest common subsumer of two classes in lexical structure of WordNet. IC, ie.. information content, is the probability encounter a word in specific corpus. SemCor Corpus adopted to count the word frequency here. We take the maximum among all base classes to obtain

---

CON
bind
Foor
horsa
honse
COW
(a) TFA
(b) Association
(c) Discrimination
Figure 3: t-SNE here shows the distribution of feature after FC2/FC22 from 200 randomly selected images
PASCAL VOC, 'horse" and 'dog' base classes, 'cow" and 'bird' are novel classes, respectively. The feature space learned by FADI has a more compact intra-class structure and larger inter-class separability.
the associated base class C where j + means the base class Cj is assigned to the novel class C.N.
G- f argmax CCn C.
jE|Ca]
To this end, novel class set C is associated with subset of base class BBNN C C
Feature Distribution Alignment After obtaining the associated base class for each novel class, given a sample z of novel class O. it is associated with pscudo label v the assigned base class Cj We design a pseudo label training mechanism to directly align the feature distribution of the novel class with the assigned base class, as follows.
min Ldoly, f(z, Wol.), where z; = g( Wira): Waxsn): WIN Gase
where W means the weights are frozen. Thus, f(: W cls and o(:; W pre) indicate the classifier (one fo layer) and the feature extractor (main network structures) with frozen weights and are pre-trained on base classes, and g(:; W.Y means an intermediate structure (one or more fe layers) to align the feature distribution classifier, this intermediate structure learns to align the feature distribution of the novel class to the associated base class. The main network structures c): W (rre knowledge from base classes.
updating the weights W By assigning pseudo labels and freezing the
also fixed keep the pre-trained
As shown in Figure 2, the regressor to reduce it to a pure classification problem. During training, we freeze all parameters except the second linear layer FC2, which means g(:; W. is a single fc layer. We then construct a balanced training set with K shots per class. is noted discard the base classes that associated with novel classes in this step. And the labels of novel classes are replaced by their assigned pseudo labels. As result, the supervision will enforce the classifier identify samples of the novel class C.² the assigned base class C, which means the feature representation of novel classes before the classifier gradually shifts toward their assigned base classes. As shown in Figure 3b, the t-SNE [25] visualization confirms the effectiveness of our distribution alignment. After the association step, the feature distribution of two associated pairs ("bird" and "dog"; "cow" and "horse") are well aligned.
the same Rol head structure of Faster R-CNN [21]. but remove
3.3 Discrimination Step
As shown in Figure 3b, after the association step, the feature distribution each novel class is aligned with the associated base class. Therefore, this novel class will have a compact intra-class distribution and naturally distinguishable from other classes. However, association step inevitably leads to confusion between the novel class and its assigned base class. To tackle this problem, we introduce a discrimination step that disentangles the classification branches for base and novel classes. set- set-specialized margin loss is further applied to enlarge the inter-class separability.
5

---

Disentangling Given a training sample 2; with label Yi, we disentangle the classification branches for base and novel classes as follows,
min Cots (Vi, (p I),)), where
W W. W.
(4)
pd = f(g{q W oo W els). p.ps).pp .p'' = f(o(q: W W 9 6 W pre)
cds) are the classifiers for base and novel classes, respectively. g(,, W origin) g(, Was asso) are the last fo layer with frozen weights for base and novel classes, respectively. As shown in Figure 2, we disentangle the classifiers and the last fc layers (FC2 and FC2 ) for base and novel classes. FC2 and FC2' load the original weights Wriggnn that are pre-traines with base classes
where I (5 I(: W.x)
and the weights WiN. assso after association step, respectively. They frozen in the discrimination step to keep their specific knowledge for base and novel classes. Therefore, FC2 and FC2' are suitable to deal with base classes and novel classes, respectively. We attach the base classifier [(;; Wat to FC2, classifier. The novel
and the novel classifier {(: W to FC2'. The base classifier is classifier a ((CN) + 1)-way classifier since we empirically let the novel classifier be also responsible for recognizing background class Co- The prediction pd and p from these two branches will be concatenated to yield the final (JCBl + |Cx| + 1)-way prediction [p p='
Set-Specialized Margin Loss Besides disentangling, we further propose set-specialized margin loss to alleviate the confusion between different classes. Different from previous margin losses [19, 5] that directly modify the original CE loss, introduce margin loss an auxiliary loss for the classifier. Given an i-th training sample of label yi. we adopt cosine similarity to formulate the logits prediction, which follows the typical conventions in few-shot classification and face recognition [27].
T T-X W W.
e
Py.
Sy.
(5)
x
Σj= eP
where Wi is the weight of the classifier, x is the input feature and 7 is the temperature factor. We try to maximize the margin of decision boundary between Cy and any other class Cjotsn as follows,
Ly
log(((s, - -s)) + e),
(6)
j'm 1.99
where Sy. and sy are classification scores on class Cy, and Cj.tt and is small number (1e-?) to keep numerical stability.
In the scenario of few-shot learning, there exists an inherent bias that the classifier tends to predict higher scores base classes, which makes the optimization of margin loss on novel classes becomes more difficult. And the number of background (negative) samples dominates the training samples, thus we may suppress the margin loss on background class Co.
Towards the aforementioned problem, is necessary to introduce the set-specialized handling of different set of classes into the margin loss. Thanks to adopting margin loss an auxiliary benefit, our design can easily enable set-specialized handling of different sets of classes by simply re-weighting the margin loss value:
Lm Σ a- Cm, +
Σ B. + (1BEEEN)
Σ 7 Lms (1),=C9)
where a, f are hyper-parameters controlling the margin of base samples, novel samples and negative samples, respectively. Intuitively, B is larger than a because novel classes are more challenging, and is a much smaller value to balance the overwhelming negative samples. Finally, the loss function of the discrimination step is shown as in Eq. 8
= + L. 2 Lreg:
where Lots is a cross-entropy loss for classification, Greg is a smooth-L1 loss for regression, and f.m is the proposed set-specialized margin loss. Since our margin loss increases the gradients on the classification branch, we scale Freg by a factor to keep the balance of the two tasks. The overall loss takes the form of multi-task learning to jointly optimize the model.

---

Novel Split
Novel
Novel Split
Rackhone
5
2
10
5
LSTD [2]
VGG-16
8.2
1.0
12.4
29.1
38.5
11.4
3.8
5.0
15.7
31.0
12.6
8.5
15.0
27.3
36.3
YOLOv2-ft
6.6
10.7
12.5
24.8
38.6
12.5
4.2
16.1
33.9
13.0
15.0
32.2
TFSRW [13]
YOLO V2
14.8
15.5
26.7
33.9
47.2
15.7
15.3
22.7
30.1
40.5
21.3
25.6
28.4
42.8
45.9
MesaDee [30]
17.1
19.1
28.9
35.0
48.8
18.2
20.6
25.9
20.1
22.3
27.9
41.9
42.9
RepMMt
Inception3
26.1
32.9
34.4
38.6
41.3
17.2
22.1
23.4
28.3
35.8
27.5
31.1
31.5
34.4
37.2
FRCN-Fi [30]
45.6
15.3
26.2
39:
9.8
45.1
FRCN+FFN-D [25]
FRCN-RIDI
8.2
20.3
29.0
40.1
45.5
13.4
20.6
28.6
32.4
38.8
19.6
20.8
28.7
-1..2
42.1
MetaDet [30]
23.1
Mota R-CNN [35]
19.9
25.5
35.0
45.7
51.5
10.4
19.4
29.6
34.8
45.4
14.3
18.2
27.5
41.2
48.1
TTA w/ fo [28]
36.8
29.1
43.6
55.7
57.0
18.2
29.0
33.4
35.5
39.0
27.7
33.6
42.5
48.7
50.2
THA [28]
39.8
36.1
44.7
55.7
56.0
23.5
26.9
34.1
35.1
39
30.8
42.8
49.5
49.8
MPSR [3]]
FRCN-RI01
41.7
51.4
55.2
61.8
24.4
39.2
39.9
47.8
35.6
42.3
48.0
49.7
[38]
47.8
55.2
35.3
46.9
46.4
FSCE [23]
14.2
43.8
51.4
61.9
63.4
27.3
29.5
43.5
44.2
50.2
37.2
41.9
47.5
54.6
58.5
FADI (Ours)
50.3
54,8
54.2
59.3
63.2
30.6
35.0
40.3
42.8
48.0
45.7
49.7
49.1
55.0
59.6
Table Performance (novel AP50) on PASCAL VOC dataset.
denotes meta-learning lew--eean-nabeeedd based methods.
nAp
nAP50
nAP75
nAP
nAP75
shot
Method
TFA
FADI
TFA
FADI
TEA
FADI
10
30
10
30
I
3.4
5.7
5.8
10.4
3.8
6.0
*FSRW [13]
5.6
4.6
7.6
2
4.6
7.0
8.3
13.1
4.8
7.0
'MetaDet [30]
7.1
11.3
5.9
10.3
3
6.6
8.6
15.8
6.5
8.3
Meta R-CNN [35]
8.7
12.4
6.6
10.8
MPSR [31]
9.8
14.1
9.7
14.2
5
8.3
10.1
15.3
18.6
8.0
9.7
SRR-FSD [38]
11.3
14.7
9.8
13.5
10
10.0
12.2
19.1
22.7
9.3
11.9
FSCE [23]
11.9
16.4
10.5
16.2
30
13.7
16.1
24.9
29.1
13.4
15.8
Ours (FADI)
12.2
16.1
11.9
15.8
(a) Comparison with baseline
(b) Comparison with latest methods.
Table 2: Performance on MS COCO dataset.
i denotes meta-learning-bart based methods. nAP means novel AP.
4
Experiments
4.1 Datasets and Evaluation Protocols
We conduct experiments both PASCAL VOC (07 12) [7] and MS COCO [18] datasets. To ensure fair comparison, we strictly follow the data split construction and evaluation protocol used in [13, 28, 23]. PASCAL VOC contains 20 categories, and we consider the same 3 base/novel splits with TFA [28] and refer them as Novel Split 1, 2, 3. Each split contains 15 base categories with abundant data and 5 novel categories with K annotated instances for K 2, 5. 10. We report AP50 of novel categories (nA P50) on VOCO7 test set. For MS COCO, 20 classes that overlap with PASCAL VOC are selected novel classes, and the remaining 60 classes are base ones. Similarly, evaluate our method on shot 1, 2, 3, 5, 10, 30 and the standard COCO-style metric is adopted.
4.2 Implementation Details
We implement our methods based on MMDetection [3]. Faster-RCNN [21] with Feature Pyramid Network [17] and ResNet-101 [12] are adopted as base model. Detailed settings are described the supplementary material.
4.3 Benchmarking Results
Comparison with Baseline Methods To show the effectiveness our method, we first make detailed comparison with TFA since our method is based on it. As shown in Table 1, FADI outperforms TFA by large margin in any shot and split on PASCAL VOC benchmark. To be specific FADI improves TFA by 10.5, 18.7, 9.5, 3.6, 7.2 and 7.1, 8.1, 6.2, 7.7, 8.9 and 14.9, 14.9, 6.3, 5.5, 9,8 for K= 3, 5, 10 on Novel splitl, split2 and split3. The lower the shot, the more difficult to learn discriminative novel classifier. The significant performance gap reflects our FADI can effectively alleviate such problem even under low shot, i.e., <= 3. Similar improvements can be observed on the challenging COCO benchmark. As shown in Table 2, we boost TFA by 2.3, 2.4, 2.0, 1.8, 2.2, 2.4

---

Association
Disentangling
Margin
nAP50
Margin
nAP50
5
TFA
41.3
K
41.3
46.3
53.7
CosFace [27]
38.9
X
42.4
46.8
55.2
ArcFace
37.9
X
/
X
42.2
47.3
54.1
/
X
44.9
50.3
56.8
CosFace (novel)
44.2
X
X
46.3
48.8
56,4
ArcFace (novel)
44.3
50.3
54.2
59.3
Ours
46.3
Table Effectiveness different components FADI.
Table 4: Comparison of different
gin loss on the TFA baseline model.
base / novel
bird
bus
cow
motorbike
sofa
nAP50
random
person
boat
horse
aeroplane
sheep
39.6
human
aeroplane
train sheep bicycle
chair
44.1
visual
dog car sheep
horse person
chair
43.3
top2
dog
tv
diningtable
41.2
topl
horse
train
horse
bicycle
chair
44.3
topl w/o dup
dog
train
horse
bicycle
chair
44.9
Table 5: Comparison of different assign policies. Set-specialized margin loss is not adopted in this table.
for K=1, 3, 5, 10, 30. Besides, also report nAP50 and nAP75, a larger gap can be obtained under IoU threshold 0.5 which suggests FADI benefits more under lower IoU thresholds.
Comparison with State-of-the-Art Methods Next, we compare with other latest few-shot meth- ods. As shown in Table 1, our method pushes the envelope of current SOTA by large margin in shot 2, 3 for novel split and 3. Specifically, we outperform current SOTA by 2.5, 4.3, 2.8 and 5.6, 7.8, 1.6 for K 1, 2,3 on novel splitl and 3, respectively. As the shot grows, the performance of FADI slightly behind FSCE [23], we conjecture by unfreezing more layers the feature extractor, the compact feature space for novel classes it exploits less base knowledge, and can better represent the real distribution than the distribution imitated by our association. However, it is not available when the shot is low the learned distribution will over-fit training samples.
model can learn
4.4 Ablation Study
In this section, we conduct thorough ablation study of each component of our method. We first analyze the performance contribution of each component, and then we show the effect of each component and why they work. Unless otherwise specified, all ablation results reported Novel Split of Pascal VOC benchmark based on our implementation TFA [28].
Component Analysis Table shows the effectiveness of each component, i.e., Association, Disen- tangling, and Set- SpecSapcill Margin Loss in our method. It is noted that when we study association without disentangling, we train a modified TFA model by replacing the FC₂ with FC2' after the association step. Since the association confuses the novel and its assigned base class, the performance of only applying association is not very significant. However, when equipped with disentangling, can significantly boost the nAP50 by 3.6, 4.0, 3.1 for ==1, 3, 5, respectively. The set-specialized margin loss shows it is generally effective for both baseline and the proposed 'association + disentangling" framework. Applying margin loss improves 'association disentangling' by 5.4, 3,9, 2.5. With all 3 components, our method totally achieves a gain of 9.0, 7.9, 5.6.
Somantic-Guided Association The assigning policy key component in the association step. To demonstrate the effectiveness of our semantic-guided assigning with WordNet [20]. we explore different assign policies. The results are shown Table 5. Random means we randomly assign base class to a novel class. Human denotes manually assigning based on human knowledge. Visual denotes associating base and novel classes by visual similarity. Specifically, we regard the weights of the base classifier as prototype representations of base classes. As a result, the score prediction of novel instances on base classifier can be viewed as the visual similarity. Top1 and top2 mean the strategies that assign each novel class to the most second similar base classes by Eq. 1. In such

---

TFA
301
plant
dreap
Association
Discrimination
com
300
Figure 4: Score confusion matrices of different methods Pascal VOC novel splitl. The element in i-th row, j-th column represents for samples of novel class i, score prediction on class j. Brighter colors indicate higher scores. If class and j are the same, this indicates a more accurate score prediction. Otherwise, indicates a heavier confusion. The font color of classes represents the association relations, egg. the associated pains 'hird' and 'dog' have the same font color blue.
Novel Split1
Novel Split2
Novel Split3
Metric
Visual
49.3
56.4
22.5
37.2
39.3
31.8
43.1
50.7
Semantic
44.9
50.3
56.8
26.1
38.5
40.1
37.1
45.0
51.5
Table 6: Comparison of visual and semantic similarity.
Figure S: Examples of co-occurance
cases, one base class may be assigned to two different novel classes ("horse" is assigned to "bird" and "cow"), we remove such duplication by taking the similarity as the priority of assigning. Specifically, the base and novel classes with the highest similarity will be associated, and they will be removed from the list of classes to be associated. Then we rank the similarity the remaining classes and choose the new association. We learn that top1 is better than random and top2 by 4.7 and 3.1, which suggests semantic similarity has a strong implication with performance. By removing the duplication, further obtain 0.6 gain.
Pot-Specialized Margin Loss Table 4 compares our margin loss with Areface [5] and CosFace [27]. can be shown that directly applying these two margin losses will harm the performance. But the degeneration of performance
be reserved by only applying to samples of novel classes. This rescues Areface from 37.9 to 44.3, Cosface from 38.9 to 44.2. Nevertheless, they are still inferior to margin loss by 2.0. Detailed typer-parameter study is described in the supplementary materials.
Complementarity between Association and Discrimination Figure 4 shows the score confusion matrices of different methods. We can see that there exists an inherent confusion between some novel and base classes, c.g. in the left top figure, "cow" is confused most with "sheep" and then "horse". However, our association biases such confusion and enforces "cow" be confused with its more semantic similar class "horse", which demonstrates the association step can align the feature distribution of the associated pairs. On the other hand, thanks the discrimination step, the confusion incurred by association is effectively alleviated and overall it shows less confusion than TFA (the second column of Figure 4). Moreover, our FADI yields significantly higher score predictions than TFA, which confirms the effectiveness of disentangling and set-specialized margin loss.
Superiority of Semantic Similarity over Visual Similarity Table 5 demonstrates semantic sim- ilarity works better than visual similarity. Here the weights of the base classifier as prototype representations of base classes. Thus, we take the score prediction of novel instances on base clas- sifier as the visual similarity. However, we find sometimes can be misleading, especially when novel instance co-occurrent with base instance, e.g., 'cat" sits on a 'chair', 'person' rides 'bike' as shown Figure 5. Such co-occurrence deceives the base classifier that 'cat' is similar to 'chair' and "bike" is similar to 'person', which makes the visual similarity not reliable under data scarcity scenarios. shown Table 6, when the shot grows, the performance gap between semantic and visual can be reduced by more accurate visual similarity measurement.

---

5 Conclusion
In this paper, we propose Few-shot object detection via Association and DIscrimination (FADI). In the association step. to learn a compact intra-class structure, we selectively associate each novel class with a well-trained base class based on their semantic similarity. The novel class readily learns to align its intra-class distribution to the associated base class. In the discrimination step, to ensure the inter-class separability, we disentangle the classification branches for base and novel classes, respectively. set-specialized margin loss further imposed to enlarge the inter-class distance. Experiments results demonstrate that FADI is concise yet effective solution for FSOD.
Acknowledgements. This research was conducted in collaboration with SenseTime. This work is supported by GRF 14203518 18, ITS/431/18FX, CUHK Agreement TS1712093, Theme-base Research Scheme 2020/21 (No. 741--00/20- R), NTU NAP, RIE2020 Industry Alignment Fund- Industry Collaboration Projects (IAF-ICP) Funding Initiative, and Shanghai Committee of Science and Technology, China (Grant No. 20DZ1 100800.
References
[1] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In IEEF
Conference on Computer Vision and Pattern Recognition, 2018. 8.
[2] Hao Chen, Yali Wang, Guoyou Wang, and Yu Qiao. Lstd: A low-shot transfer detector for object detection.
In AAAI Conference on Artificial Intelligence, 2018. 7
[3] Kai Chen, Jiaqi Wang, Jiangmiao Pang. Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng. Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng. Chenchen Zhu, Tianheng Cheng. Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint
arXiv: (ifi:/900.07155.. 2019. 7, 12
[4] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. In International Conference on Learning Representations, 2019. 3
[5] Jiankang Deng. Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2019. 3. 6, 8, [6] Guncet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot
image classification. arXiv preprint arXiv: / 1909.02729 2019. 3
[7] M. Everingham. L. Van Gool. C. K. I. Williams. J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303-335 June 2010. 2, 7,
[8] Qi Fan, Wei Zhuo, Chi-Keung Tang, and Yu-Wing Tai. Few-shot object detection with attention-rpn and multi-relation detector. In IEEE Conference on Computer Vision and Pattern Recognition, 2020. [9] Chelsea Finn. Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leariing for fast adaptation of deep networks. In International Conference on Machine Learning, 2017. 3
[10] Bharath Hariharan and Ross Girshick. Low-shot visual recognition by shrinking and hallucinating features.
In IEEE International Conference on Computer Vision, 2017. 3
[11] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In IEEE International
Conference on Computer Vision, 2017.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[13] Bingyi Kang, Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, and Trevor Darrell. Few-shot object detection
via feature reweighting. In JEFE International Conference on Computer Vision, 2019. 2, 3, 7
[14] Leonid Karlinsky, Joseph Shtok, Sivan Harary, Eli Schwartz, Amit Aides, Rogerio Feris, Raja Giryes, and Alex Bronstein. Repmet: Representative based metric learning for classification and few-shot object detection. In JEEE International Conference on Computer Vision, 2019, 2, 3.7
[15] Kwonjoon Lec, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Mcta-Icarning with differ- entiable convex optimization. In JEEE Conference on Compuder Vision and Pattern Recognition, 2019.
[16] Dekang Lin et al. An information theoretic definition of similarity, In International Conference on Machine Learning. 1998. 4
[17] Tsung-Yi Vi Lin, Piotr Dollár, Ross B Girshick, Kaiming He, Bharath Hariharan, and Serge J Belongie. Feature pyramid networks for object detection. In IEEE Conference on Computer Vision and Pattern Recognition. 2017.
[18] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, 2014. 2, 7, 12
[19] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li. Bhiksha Raj. and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2017. 3,
10

---

[20] George A Miller. Wordnet: a lexical database for english. Communications of the ACM. 4.8
[21] Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real- time object detection
with region proposal networks. In Advances in Neural Information Processing Systems, 2015. 5.
[22] Jake Snell, Kevin Swersky, and Richard S 7emel. Prototypical networks for few-shot learning. In Advances
in Neural Information Processing Systems, 2017.
[23] Bo Sun, Banghuai Li, Shengcai Cai, Ye Yuan, and Chi Zhang. Fsce: Few-shot object detection via
contrastive proposal encoding. In JEEE Conference on Computer Vision and Pattern Recognition, 2021. 2,
3, 7,
[24] Flood Sung, Yongxin Yang. Li Zhang. Tao Xiang, Philip HS Torr, and Timothy Hospedales. Learning to compare: Relation network for few-shot learning. In IEEE Conference on Computer Vision and Pattern Recognition, 2018. 3
[25] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning
reseurch. 2008. 5
[26] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching
networks for one shot learning. In Advances in Neural Information Processing Systems, 2016. 3
[27] Hao Wang. Yitong Wang. Zheng Zhou, Xing Ji, Dihong Gong. Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In IEEE Conference on Computer Vision and
Pattern Recognition. 2018. 3, 8,
[28] Xin Wang. Thomas F. Huang. Trevor Darrell, Joseph E Gonzalez, and Fisher Yu. Frustratingly simple few-shot object detection. In International Conference on Machine Learning, 2020. 2, 3, 7,
[29] Yu-Xiong Wang. Ross Girshick, Martial Hebert, and Bharath Hariharan. Low-shot learning from imaginary
data. In IEEE Conference on Computer Vision and Puttern Recognition, 2018. 3
[30] Yu-Xiong Wang. Deva Ramanan, and Martial Hehert. Meta-learning to detect rare objects. In IEFE
Conference on Computer Vision and Pattern Recognition, 2019. 7
[31] Jiaxi Wu, Songtao Liu, Di Huang, and Yunhong Wang. Multi-scale positive sample refinement for few-shot
object detection. In Eurupean Conference on Computer Vision, 2020. 2, 3.
[32] Yang Xiao and Renaud Marlet. Few-shot object detection and viewpoint estimation for objects in the wild.
In European Conference Computer Vision, 2020. 3.
[33] Chen Xing. Negar Rostamzadeh, Boris Oreshkin. and Pedro OO Pinheiro. Adaptive cross modal few-shot
learning. Advances in Neural Information Processing Systems, 2019.
[34] Caixia Yan, Qinghua Zheng, Xiaojun Chang. Minnan Luo, Chung-Hsing Ych, and Alexander G Hauptman. Demantics-preservinn graph propagation for zero-shot object detection. IEEE Transactions on Image Processing, 2020.
[35] Xiaopeng Yan, Ziliang Chen, Anni Xu, Xiaoxi Wang, Xiaodan Liang, and Liang Lin. Meta r-cnn: Towards general solver for instance- low-shot learning. In JEEE International Conference on Computer Vision,
2019. 2, 3. 7
[36] Yukuan Yang. Fangyu Wei, Miaojing Shi, and Guoqi Li. Restoring negative information in few-shot object detection. In Advances in Neural Information P Processing Systems. 2020. 3
[37] Ze Yang. Yali Wang, Xianyu Chen, Jianzhuang Liu, and Yu Qiao.
tackling object
confusion for few-shot detection. In AAAI Conference on Artificial Intelligence, 2020. 3
[38] Chenchen Zhu, Fangyi Chen, Uzair Ahmed, and Marios Savvides. Semantic relation reasoning for shot- stable fow-shot object detection. In IEEE Conference Computer Vision and Pattern Recognition, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Appendix 1.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [No] We will release
the code to ensure strict reproducibility upon the paper accepted.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4. Appendix 2 and Appendix 3.
11

---

(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix 2.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] Pascal VOC [7], MS
COCO [18], MMDetection [3]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "zF9j-CTbNJQ", "reviewer": "Reviewer_zrXC", "summary": "This paper proposes a two-step fine-tuning framework for few-shot object detection, where the first stage is an association step, associating each novel class with a base class according to their semantic similarity, and the second stage is a discrimination step, which ensure the separability between the novel classes and associated base classes. Experiments show the effectiveness of the proposed method. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a new framework for few-shot object detection and achieves good results on some indicators. Generally, the paper is well organized and written.\n\nHowever, there are obvious limitations with this paper: \n1. Feature distribution alignment module is interesting and novel. However, the proposed set-specialized margin loss has limited innovation. The loss is very similar to cos similarity, which is also used in TFA. Each line of W in Eq.5 can be regraded as a prototype of a category. The main innovation of the proposed loss function lies in that the weights of the novel classes are increased by hyperparameter $\\beta$, and the weight of the background class is reduced by hyperparameter $\\gamma$. It's more like a technical trick. In line 198-201, the authors explained the motivation of reweighing. As can be seen from Appendix C, when $\\alpha/\\beta $ increases, the AP50 improves little (only from 49.6 to 50.3). Similarly, for the adjustment of gamma, the effect on the result is relatively small. Therefore, the validity of this technique is questionable. Moreover, the authors did not do experiment for $\\gamma$ = 1 in Table 4 (b) of Appendix C.\n\n2. In experiments, comparisons with the latest SOTAs are missed. Following are \nsome papers in CVPR2020 have not been compared:\n[1] Fan, Qi, et al. \"Few-shot object detection with attention-RPN and multi-relation detector.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n[2] Yang, Yukuan, et al. \"Restoring Negative Information in Few-Shot Object Detection.\" Advances in Neural Information Processing Systems 33 (2020): 3521-3532.\n[3] Yang, Ze, et al. \"Context-transformer: tackling object confusion for few-shot detection.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 07. 2020.\n\nIn addition, comparing the papers in CVPR 2021, the performance of the proposed method is not promising:\n[4] Zhang, Weilin, and Yu-Xiong Wang. \"Hallucination Improves Few-Shot Object Detection.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n[5] Li, Yiting, et al. \"Few-Shot Object Detection via Classification Refinement and Distractor Retreatment.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n[6] Fan, Zhibo, et al. \"Generalized Few-Shot Object Detection without Forgetting.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n[7] Li, Aoxue, and Zhenguo Li. \"Transformation Invariant Few-Shot Object Detection.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n[8] Zhang, Lu, et al. \"Accurate Few-Shot Object Detection With Support-Query Mutual Guidance and Hybrid Loss.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n[9] Hu, Hanzhe, et al. \"Dense Relation Distillation with Context-aware Aggregation for Few-Shot Object Detection.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n[10] Li, Bohao, et al. \"Beyond Max-Margin: Class Margin Equilibrium for Few-shot Object Detection.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n3. Line 181:  Where is $\\widetilde{W}^B_{asso}$ from? As I understand, in the association step, we can obtain $\\widetilde{W}^N_{asso}$, which is the intermediate structure that follows $\\phi(;\\widetilde{W}^B_{pre})$ (feature extractor). $\\widetilde{W}^N_{asso}$ is used to perform feature distribution alignment for novel classes, but for base classes, it doesn’t need to perform feature distribution alignment. So, what does $\\widetilde{W}^B_{asso}$ mean? In line 181-182, it’s said that it comes from the association step, but I cannot find the variable in the association step.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. If the semantic similarity between the novel class and all base class is small, there may be some problems with the association process. It is possible to choose the most similar one among the many dissimilar base classes. However, the selected base class may be not similar to the novel class. On the other hand, it is possible that the similarity between a novel class and multiple base classes is very high. In this case, there may be some problems when aligning it to one of the base classes.\n\n2. In the association step, there may be a visual semantic gap. That is, some classes are semantically similar, but not visually similar. This problem is also common in zero-shot learning and zero-shot object detection [1]. However, the authors didn’t consider this issue.\n\n[1] C. Yan, Q. Zheng, X. Chang, M. Luo, C. -H. Yeh and A. G. Hauptman, \"Semantics-Preserving Graph Propagation for Zero-Shot Object Detection,\" in IEEE Transactions on Image Processing, vol. 29, pp. 8163-8176, 2020, doi: 10.1109/TIP.2020.3011807.\n\n3. The experimental setting of this paper is different from the previous FSOD works. This work introduces the semantic information of the category, which is equivalent to having additional information input. There are some works in few-shot learning [2], and the introduction of semantic information can improve performance by a large margin. However, the performance improvement brought by semantic information in this paper is limited (in Tab.3, less than 1 point in 1/3-shot)，this makes the effectiveness of the method questionable. \n\n[2] Chen Xing, Negar Rostamzadeh, Boris N Oreshkin and Pedro O Pinheiro, \"Adaptive cross-modal few-shot learning\", Advances in Neural Information Processing Systems, pp. 4848-4858, 2019.\n", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Qb0ebxyZMVI", "reviewer": "Reviewer_oVR6", "summary": "The paper proposes a novel approach for few-shot object detection by first assigning a novel (few-shot) class to a base category (Association) and then disentangling the classification of base and novel classes (Discrimination). After the first step of base training, the proposed approach assigns each novel class to a base class using the WordNet hierarchy. Then, the method aligns the feature distribution of the novel class to the associated base class. Finally, the method proposes to disentangle the classification branches of base and novel classes to remove confusion between base and novel classes. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed approach is highly intuitive and has been well described by the authors. I believe that this idea is novel and is a strong contribution to the field. The authors have also demonstrated the effectiveness of the proposed approach through strong results on the PASCAL VOC and MSCOCO datasets in the few-shot setting. \n\nI have a few questions/suggestion about the paper which the authors should address in their response. \n\n1. In Table 1, the authors show good improvements for low-shot learning in Split 1 and Split 3. Why is the performance not so good for 5 and 10 shots for Split 1? 5-shot seems to be particularly low compared to FSCE. Why is this the case? Was the performance of FSCE somehow artificially too high in this case or is the performance of FADI artificially too low? Further, why is there no similar improvement for Split 2? How is Split 2 different from Splits 1 and 2?\n\n2. For Table 5, do the same observations hold for larger datasets like MSCOCO? Do these observation change with change in dataset?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative social impact of their work but only in the supplementary material and not in the main text of the paper. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "30NTQEqL80", "reviewer": "Reviewer_VAwp", "summary": "\nThe paper proposes a method for few shot object detection. The key idea is to leverage the similarity of a novel object class with an existing base class, and then train a local classifier for better discrimination. The class similarity is based on semantic word similarity between the classes’ textual names and the new class to base class mapping is an injective mapping. Once the assignment is calculated the feature network is trained so that the features of the new class align with its corresponding base class. Finally further discriminator layers are used which are then trained to classify base and new classes distinctly. Empirical results are shown on PASCAL VOC 2007 2012 and COCO datasets, where the proposed method outperforms existing methods especially in very low shot settings.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main motivation seems to be that if you fine-tune with the less amount of data, then the local optima of the resulting optimization might not be good. So the idea is to constrain the optimization region with priors. The first being that the feature network should be highly biased to keep similar classes in the same locality in both base and new classes. Second one being that the classifiers for the similar base and novel classes should be in some sense trained locally.\n\nThe colors used in fig4 could be better, I had to look carefully to see that book and cow are two different colors and not different weights of different colors (first reflex is to associate light color with small weight, on which I had to remind myself that the association is binary, right?). Use contrasting colors. And perhaps for the non matched classes have highlights or underlines or some other decoration to make it easy to perceive.\n\nAlso in fig4 score confusion matrix is shown, but what exactly is score confusion is not defined? It is also not very clear why the left column on shows base classes in the x axis which the right col has both base and new ones.\n\nIn table 5 association comparison with visual features derived similarity should also be compared with the semantic text similarity. It would be interesting to see how much worse is that.\n\nThe tab6 argument seems weak as in one shot situation, we would expect the variance to be very high, and the numbers to flicker a lot depending on the splitting. With even a slight increase in number of examples the visual and textual similarities seem to be similar.\n\nThe AP is reported only on novel classes (nAP). As experienced with zero and few shot classification works, there is usually a tradeoff between novel class only performance and whole set performance. I understand that practitioners may be only interested in new class performances, but academically we should be aware of how the performances are suffering for old classes, and all classes taken together. Since those metrics should be easy to compute with the new model, I would strongly encourage them to be reported (even if only for the current method, as the older methods might not have reported them. I see that TFA has reported that on base and novel separately). Since the method here is advocates strong local learning, it might be alleviating confusion for new classes, but might be increasing the same for old classes. A discussion would also be appreciated.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "2jjlxj-GLUl", "reviewer": "Reviewer_DsGJ", "summary": "Durning ﬁne-tuning of few-shot object detection, a novel class may implicitly leverage the 8 knowledge of multiple base classes to construct its feature space, which induces a scattered feature space, hence violating the inter-class separability. To overcome these obstacles, we propose a two-step ﬁne-tuning framework, Few-shot object detection via Association and DIscrimination (FADI), which builds up a discriminative feature space for each novel class with two integral steps. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am in general positive towards this paper, as the method has convincing motivation, and is presented very clearly. \n\nI still have a few comments: \n\nAuthors claim that existing SOTA few-shot detection works all employ a finetuning strategy, this is not appropriate. There are still several works focusing on the meta-learning mode, for instance, RepMet [13] and NP-RepMet [NIPS2020]. \n\nFor the class similarity measure, authors choose WordNet over over visual similarity.  From my viewpoint, visual similarity also has advantages that WordNet does not. For instance, an airplane and a bird can only be similar visually.  \n\nFor the feature distribution alignment, what's the motivation to implement it this way ? Have you tried some other ways, for instance, gradient reversal layer ?  \n\nThe method performs SOTA on low-shots in most scenarios, yet not in the second split of VOC. Any discussions and observations on this ? \nAlso, have authors tried to integrate their methods into FCSE to see if they could further achieve SOTA on rather higher shots ? \n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours "}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an alternative to fine-tuning for few-shot object detection, by instead associating each new class to an existing base class and then training with a max margin loss between this new class from the base class. The idea is simple, its implementation probably still leaves some performance on the table (e.g. visual similarity vs. WordNet similarity). The experimental validation is strong, in particular in term of AP on new classes on standard benchmarks.

The reviewers agree that the paper is mostly clear and convincing. The authors addressed some of the limitations of the paper in the author response, in particular in term of comparisons with the literature. I recommend acceptance as a poster.
Decision: Accept (Poster)