Few-Shot Object Detection via Association and DIscrimination

1 Introduction . Deep learning has achieved impressive performance on object detection [ 21 , 11 , 1 ] in recent years . However , their strong performance heavily relies on a large amount of labeled training data , which limits the scalability and generalizability of the model in the data scarcity scenarios . In contrast , human visual systems can easily generalize to novel classes with only a few supervisions . Therefore , great interests have been invoked to explore few-shot object detection ( FSOD ) , which aims at training a network from limited annotations of novel classes with the aid of sufficient data of base classes . BCorresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Various methods have since been proposed to tackle the problem of FSOD , including metalearning [ 13 , 35 , 32 ] , metric learning [ 14 ] , and fine-tuning [ 28 , 31 , 23 ] . Among them , fine-tuningbased methods are one of the dominating paradigms for few-shot object detection . [ 28 ] introduces a simple two-stage fine-tuning approach ( TFA ) . MPSR [ 31 ] improves upon TFA [ 28 ] via alleviating the problem of scale variation . The recent state-of-the-art method FSCE [ 23 ] shows the classifier is more error-prone than the regressor , and introduces the contrastive-aware object proposal encodings to facilitate the classification of detected objects . All these works employ a holistic fine-tuning paradigm , where the model is first trained on all base classes with abundant samples , and then the pre-trained model is fine-tuned on novel classes . Although it exhibits a considerable performance advantage compared with the earlier meta-learning methods , this fine-tuning paradigm is still imperfect . To be specific , the current design of the fine-tuning stage directly extracts the feature representation of a novel class from the network pre-trained on base classes . Therefore , a novel class may exploit the knowledge of multiple similar base classes to construct the feature space of itself . As a result , the feature space of a novel class will have an incompact intra-class structure that scatters across feature spaces of other classes , breaking the inter-class separability , hence leading to classification confusion , as shown in Figure 1a . To overcome these obstacles , we propose a two-step fine-tuning framework , Few-shot object detection via Association and DIscrimination ( FADI ) , which constructs a discriminable feature space for each novel class with two integral steps , association and discrimination . Specifically , in the association step , as shown in Figure 1b , to construct a compact intra-class structure , we associate each novel class with a well-trained base class based on their underlying semantic similarity . The novel class readily learns to align its feature space to the associated base class , thus naturally becomes separable from the remaining classes . In the discrimination step , as shown in Figure 1c , to ensure the separability between the novel classes and associated base classes , we disentangle the classification branches for base and novel classes to reduce the ambiguity in the feature space induced by the association step . To further enlarge the inter-class separability between all classes , a set-specialized margin loss is applied . To this end , the fine-tuning stage is divided into two dedicated steps , and together complement each other . Extensive experimental results have validated the effectiveness of our approach . We gain significant performance improvements on the Pascal VOC [ 7 ] and COCO [ 18 ] benchmarks , especially on the extremely few-shot scenario . Specifically , without bells and whistles , FADI improves the TFA [ 28 ] baseline by a significant margin in any split and shot with up to +18.7 mAP , and push the envelope of the state-of-the-art performance by 2.5 , 4.3 , 2.8 and 5.6 , 7.8 , 1.6 for shot K = 1 , 2 , 3 on novel split-1 and split-3 of Pascal VOC dataset , respectively . 2 Related Work . Few-Shot Classification Few-Shot Classification aims to recognize novel instances with abundant base samples and a few novel samples . Metric-based methods address the few-shot learning by learning to compare , different distance formulations [ 26 , 22 , 24 ] are adopted . Initialization-based methods [ 9 , 15 ] learn a good weight initialization to promote the adaption to unseen samples more effectively . Hallucination-based methods introduce hallucination techniques [ 10 , 29 ] to alleviate the shortage of novel data . Recently , researchers find out that the simple pre-training and fine-tuning framework [ 4 , 6 ] can compare favorably against other complex algorithms . Few-Shot Object Detection As an emerging task , FSOD is less explored than few-shot classification . Early works mainly explore the line of meta-learning [ 13 , 35 , 32 , 14 , 36 , 37 , 8 ] , where a meta-learner is introduced to acquire class agnostic meta knowledge that can be transferred to novel classes . Later , [ 28 ] introduces a simple two-stage fine-tuning approach ( TFA ) , which significantly outperforms the earlier meta-learning methods . Following this framework , MPSR [ 31 ] enriches object scales by generating multi-scale positive samples to alleviate the inherent scale bias . Recently , FSCE [ 23 ] shows in FSOD , the classifier is more error-prone than the regressor and introduces the contrastive-aware object proposal encodings to facilitate the classification of detected objects . Similarly , FADI also aims to promote the discrimination capacity of the classifier . But unlike previous methods that directly learn the classifier by implicitly exploiting the base knowledge , motivated by the works of [ 34 , 33 ] , FADI explicitly associates each novel class with a semantically similar base class to learn a compact intra-class distribution . Margin Loss Loss function plays an important role in the field of recognition tasks . To enhance the discrimination power of traditional softmax loss , different kinds of margin loss are proposed . SphereFace [ 19 ] introduces a multiplicative margin constrain in a hypersphere manifold . However , the non-monotonicity of cosine function makes it difficult for stable optimization , CosFace [ 27 ] then proposed to further normalize the feature embedding and impose an additive margin in the cosine space . ArcFace [ 5 ] moves the additive cosine margin into the angular space to obtain a better discrimination power and more stable training . However , we find these margin losses are not directly applicable under data-scarce settings as they equally treat different kinds of samples but ignore the inherent bias of the classifier towards base classes . Hence we propose a set-specialized margin loss that takes the kind of samples into consideration which yields significantly better performance . 3 Our Approach . In this section , we first review the preliminaries of few-shot object detection setting and the conventional two-stage fine-tuning framework . Then we introduce our method that tackles few-shot object detection via association and discrimination ( FADI ) . 3.1 Preliminaries . In few-shot detection , the training set is composed of a base set DB = { xBi , yBi } with abundant data of classes CB , and a novel set DN = { xNi , yNi } with few-shot data of classes CN , where xi and yi indicate training samples and labels , respectively . The number of objects for each class in CN is K for K-shot detection . The model is expected to detect objects in the test set with classes in CB ∪CN . Fine-tuning-based methods are the current one of the leading paradigms for few-shot object detection , which successfully adopt a simple two-stage training pipeline to leverage the knowledge of base classes . TFA [ 28 ] is a widely adopted baseline of fine-tuning-based few-shot detectors . In the base training stage , the model is trained on base classes with sufficient data to obtain a robust feature representation . In the novel fine-tuning stage , the pre-trained model on base classes is then fine-tuned on a balanced few-shot set which comprises both base and novel classes ( CB ∪ CN ) . Aiming at preventing over-fitting during fine-tuning , only the box predictor , i.e. , classifier and regressor , are updated to fit the few-shot set . While the feature extractor , i.e. , other structures of the network , are frozen [ 28 ] to preserve the pre-trained knowledge on the abundant base classes . Although the current design of fine-tuning stage brings considerable gains on few-shot detection , we observe that it may induce a scattered feature space on novel class , which violates the inter-class separability and leads to confusion of classification . Towards this drawback , we proposes few-shot object detection via association and discrimination ( FADI ) , which divides the fine-tuning stage of TFA into a two-step association and discrimination pipelines . In the association step ( Sec . 3.2 ) , to construct a compact intra-class distribution , we associate each novel class with a base class based on their underlying semantic similarity . The feature representation of the associated base class is explicitly learned by the novel class . In the discrimination step ( Sec . 3.3 ) , to ensure the inter-class separability , we disentangle the base and novel branches and impose a set-specialized margin loss to train a more discriminative classifier for each class . 3.2 Association Step . In the base training stage , the base model is trained on the abundant base data DB and its classifier learns a good decision boundary ( see Figure 1 ) to separate the whole decision space into several subspaces that are occupied by different base classes . Therefore , if a novel class can align the feature distribution of a base class , it will fall into the intra-class distribution of the associated base class , and be naturally separable from the other base classes . And if two novel classes are assigned to different base classes , they will also become separable from each other . To achieve this goal , we introduce a new concept named association , which pairs each novel class to a similar base class by semantic similarity . After then , the feature distribution of the novel class is aligned with the associated base class via feature distribution alignment . Similarity Measurement In order to ease the difficulty of feature distribution alignment , given a novel class CNi and a set of base classes C B , we want to associate CNi to the most similar base class in CB . An intuitive way is to rely on visual similarity between feature embeddings . However , the embedding is not representative for novel classes under data-scarce scenarios . Thus , we adopt WordNet [ 20 ] as an auxiliary to describe the semantic similarity between classes . WordNet is an English vocabulary graph , where nodes represent lemmas or synsets and they are linked according to their relations . It incorporates rich lexical knowledge which benefits the association . Lin Similarity [ 16 ] is used to calculate the class-to-class similarity upon WordNet which is given by : sim ( CNi , C B j ) = 2 · IC ( LCS ( CNi , CBj ) ) IC ( CNi ) + IC ( C B j ) , ( 1 ) where LCS denotes the lowest common subsumer of two classes in lexical structure of WordNet . IC , i.e. , information content , is the probability to encounter a word in a specific corpus . SemCor Corpus is adopted to count the word frequency here . We take the maximum among all base classes to obtain the associated base class CBj→i where j → i means the base class CBj is assigned to the novel class CNi . CBj→i ← argmax j∈|CB | sim ( CNi , C B j ) . ( 2 ) To this end , novel class set CN is associated with a subset of base class CB→N ⊂ CB . Feature Distribution Alignment After obtaining the associated base class for each novel class , given a sample xNi of novel class C N i , it is associated with a pseudo label y B j of the assigned base class CBj→i . We design a pseudo label training mechanism to directly align the feature distribution of the novel class with the assigned base class , as follows . min WNasso Lcls ( yBj , f ( zNi ; W̃Bcls ) ) , where zNi = g ( φ ( xNi ; W̃Bpre ) ; WNasso ) , ( 3 ) where W̃ means the weights are frozen . Thus , f ( · ; W̃Bcls ) and φ ( · ; W̃Bpre ) indicate the classifier ( one fc layer ) and the feature extractor ( main network structures ) with frozen weights and are pre-trained on base classes , and g ( · ; WNasso ) means an intermediate structure ( one or more fc layers ) to align the feature distribution via updating the weightsWNasso . By assigning pseudo labels and freezing the classifier , this intermediate structure learns to align the feature distribution of the novel class to the associated base class . The main network structures φ ( · ; W̃Bpre ) is also fixed to keep the pre-trained knowledge from base classes . As shown in Figure 2 , we use the same RoI head structure of Faster R-CNN [ 21 ] , but we remove the regressor to reduce it to a pure classification problem . During training , we freeze all parameters except the second linear layer FC ′ 2 , which means g ( · ; WNasso ) is a single fc layer . We then construct a balanced training set with K shots per class . It is noted we discard the base classes that are associated with novel classes in this step . And the labels of novel classes are replaced by their assigned pseudo labels . As a result , the supervision will enforce the classifier to identify samples of the novel class CNi as the assigned base class CBj→i , which means the feature representation of novel classes before the classifier gradually shifts toward their assigned base classes . As shown in Figure 3b , the t-SNE [ 25 ] visualization confirms the effectiveness of our distribution alignment . After the association step , the feature distribution of two associated pairs ( `` bird '' and `` dog '' ; `` cow '' and `` horse '' ) are well aligned .

Summary:  The paper proposes a method for few shot object detection. The key idea is to leverage the similarity of a novel object class with an existing base class, and then train a local classifier for better discrimination. The class similarity is based on semantic word similarity between the classes’ textual names and the new class to base class mapping is an injective mapping. Once the assignment is calculated the feature network is trained so that the features of the new class align with its corresponding base class. Finally further discriminator layers are used which are then trained to classify base and new classes distinctly. Empirical results are shown on PASCAL VOC 2007 2012 and COCO datasets, where the proposed method outperforms existing methods especially in very low shot settings.