Generalized and Discriminative Few-Shot Object Detection via SVD-Dictionary Enhancement

1 Introduction . With the rejuvenation of deep neural networks , for object detection , many progresses [ 11 , 12 , 1 , 26 , 22 ] have been achieved . Though these methods obtain outstanding detection performances , they usually require a large number of labeled samples for training , which are labored yet expensive to collect and annotate . On the contrary , human beings are born with the ability to learn a new visual concept with only few samples . To imitate such an ability of human beings , the task of few-shot object detection ( FSOD ) [ 2 , 17 , 36 ] has been proposed , which aims to improve the detection performance for new objects that contain few annotated training samples . The main challenge of FSOD lies in how to learn generalized and discriminative object features from both abundant samples in base object categories and few samples in new object categories , which can improve the representation ability of object features and alleviate overfitting on new objects . Following the popular methods for few-shot image classification , earlier attempts [ 38 , 37 , 33 , 8 ] in FSOD utilize the meta-learning strategy [ 29 , 31 , 10 ] , whose goal is to learn detectors across tasks and then transfer to the few-shot detection task . However , compared with traditional two-stage fine-tuning based approaches [ 34 , 35 , 30 ] , the meta-learning strategy fails to effectively improve generalization ∗Corresponding Author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and discrimination of object features and leads to weak performance . The reason may be that during each training episode , meta-learning methods focus on transferability across different tasks and ignore learning of generalized and discriminative feature representations . For FSOD , the generalized representations may contain intrinsical characteristics of object features , which is beneficial for adapting knowledge from base object categories to new object categories . Meanwhile , the discriminative representations may contain certain category-related information , which is helpful for boosting the detection accuracy . Furthermore , recent research [ 3 ] has shown that from a spectral analysis perspective , the feature representations can be decomposed into eigenvectors with importance quantified by the corresponding singular values . The eigenvectors corresponding to larger singular values contribute to the generalization ability , as these eigenvectors could suppress certain variations ( e.g. , the variations of style and texture ) . Meanwhile , since the eigenvectors corresponding to relatively smaller singular values contain richer category-related information ( e.g. , the structures of objects ) , these eigenvectors are beneficial for discrimination . Therefore , in this paper , we explore employing Singular Value Decomposition ( SVD ) ( as shown in Fig . 1 ) to promote detectors to learn generalized and discriminative object features . Particularly , we propose a method named as SVD-Dictionary enhancement for FSOD . Given an input image , a backbone network is first used to extract the corresponding feature map . Then , SVD is performed on the feature map . Here , we select the eigenvectors corresponding to the first k largest singular values to compute a generalization map . And the generalization ability is enhanced by a residual operation between the generalization map and the original feature map . Next , the residual eigenvectors are used to calculate a discrimination map . Meanwhile , to further enhance discrimination , we define a codebook containing multiple codewords and employ dictionary learning [ 41 ] to capture high-level discriminative information from the discrimination map , which is good for accurate detection . Compared with most methods [ 35 , 33 , 17 ] for FSOD , our method includes two virtues . One is that during enhancing generalization , our method does not introduce extra parameters . The other is that with the help of the discrimination map and dictionary learning , our method could capture high-level discriminative information of different categories , which is conductive to reducing the data-scarce impact on new object categories . During training , we first train the model on the data-abundant base object categories . Then , the model is fine-tuned on a reconstructed training set that contains a small number of balanced training samples from both base and new object categories . Extensive experiments on two benchmarks demonstrate the superiorities of our method . The contributions of our work are summarized as follows : • To boost both the generalization and discrimination abilities , we propose to build the generalization and discrimination spaces based on the sorted singular values . • To further enhance the discrimination ability , we explore dictionary learning to capture high-level discriminative information from the discrimination map . • By plugging our method into two two-stage methods , i.e. , MPSR [ 35 ] and FSCE [ 30 ] , our method significantly improves their performances on PASCAL VOC [ 6 , 7 ] and COCO [ 20 ] . 2 Related Work . Few-shot image classification . The goal of few-shot image classification [ 29 , 24 ] is to recognize new categories with very few labeled samples . Recently , many progresses [ 5 , 32 , 42 , 40 , 13 ] have been achieved . Particularly , meta-learning [ 10 ] is a widely used method to solve few-shot classification , which aims to leverage task-level meta knowledge to help models adapt to new tasks with few labeled samples . Based on the meta-learning policy , Snell et al . [ 29 ] proposed a prototypical network to learn a metric space in which classification can be performed by computing distances to the prototype representation of each category . However , the performance of this method relies on the quality of the learned prototypes . When the training data is scarce , the learned prototypes could not represent the information of each category sufficiently , which affects the classification performance . Liu et al . [ 23 ] proposed a method of prototype rectification , which considers the intra-class bias and the cross-class bias and improves the performance significantly . Apart from these methods , more methods , e.g. , sample synthesis and augmentation , in few-shot learning can be seen in the work [ 24 ] . Whereas , these classification methods could not be directly applied to detection that requires localizing and recognizing objects simultaneously . Few-shot object detection . Towards FSOD , most existing methods [ 18 , 8 , 25 , 2 , 39 ] employ a metalearning or fine-tuning based mechanism . Particularly , Wang et al . [ 33 ] proposed a meta-learning framework to leverage meta-level knowledge from base object categories to facilitate the generation of a detector for new object categories . Based on this work [ 33 ] , Kang et al . [ 17 ] further proposed a one-stage detection architecture that contains a meta feature learner and a reweighting module . In order to alleviate the impact of complex background and multiple objects on one image , Yan et al . [ 38 ] extended Faster R-CNN [ 27 ] and Mask R-CNN [ 16 ] by proposing meta-learning over RoI ( Region-of-Interest ) features . Recently , the two-stage fine-tuning based approach ( TFA ) [ 34 ] reveals a potential for addressing FSOD . By simply fine-tuning the box classifier and regressor , this method outperforms many meta-learning based methods . Wu et al . [ 35 ] considered the impact of the scale bias on the fine-tuning process , which further improves the detection performance . Different from the above methods , in this paper , we explore enhancing both generalization and discrimination for FSOD . And we propose a method of SVD-Dictionary enhancement that combines SVD with dictionary learning . Experimental results and visualization analysis demonstrate the superiorities of the proposed method . 3 SVD-Dictionary Enhancement for FSOD . In this paper , we follow the same settings introduced in Kang et al . [ 17 ] . Concretely , there are a set of base object categories that contain abundant annotated samples and a set of new object categories that contain only few ( usually less than 30 ) annotated samples per category . The main purpose is to improve the detection performance of new object categories . 3.1 SVD Enhancement . For FSOD , generalization and discrimination are two important criteria that characterize the goodness of feature representation . Particularly , enhancing generalization is beneficial for adapting the knowledge learned from base object categories to new object categories , which alleviates the data-scarce impact on new object categories . Meanwhile , discrimination refers to the ability to separate different categories based on the learned representations . And enhancing discrimination is helpful for reducing the overfitting risk on new object categories , which improves the detection accuracy . To this end , we explore SVD to enhance both the generalization and discrimination abilities of detectors . Concretely , as shown in Fig . 2 , we adopt a widely used two-stage object detector , i.e. , Faster R-CNN [ 27 ] , as the basic detection model . Given an input image , we first employ the feature extractor , e.g. , ResNet [ 15 ] , to extract the corresponding feature map F ∈ Rm×w×h , where m , w , and h separately denote the number of channels , width , and height . Then , F is reshaped as F ∈ Rm×n , where n = w × h. SVD is used to factorize the matrix F , i.e. , F = UΣVT ∈ Rm×n , into the product of three matrices , where U ∈ Rm×m and V ∈ Rn×n are orthogonal , and Σ contains the sorted singular values along its main diagonal [ 4 ] . Since the eigenvectors corresponding to larger singular values contain more information of the original matrix F , we select the eigenvectors corresponding to the first k largest singular values to compute the generalization map G ∈ Rm×n . Next , by feat of the residual operation between G and F , the generalization ability of the extracted features is enhanced . The processes are shown as follows : G = Um×kΣk×kVTk×n , E = G + F , ( 1 ) where Um×k and VTk×n indicate that we select the first k columns and rows from the matrix U and VT , respectively . Σk×k is a diagonal matrix with the dimension k × k. E ∈ Rm×n is the enhanced matrix . Finally , E is reshaped as E ∈ Rm×w×h that is used to perform the following RPN operation . It is worth noting that in the process of enhancing generalization , we only perform the SVD operation and do not introduce extra parameters . Besides , we utilize the residual operation to obtain the output E , which strengthens the generalization ability and retains the discriminative information in the output . In the experiment , we observe that utilizing the operation of enhancing generalization improves the detection performance effectively . Next , the remaining eigenvectors and corresponding singular values are used to calculate the discrimination map D ∈ Rm×n . The processes are the same as computing G. Since the map D contains more category-related information [ 3 ] , e.g. , the structures of objects , it is helpful for enhancing the discrimination ability . Similarly , for this process , we do not introduce extra parameters , either . 3.2 SVD-based Dictionary Learning . Dictionary Learning . Based on the map D , we explore employing dictionary learning [ 41 , 14 ] to capture high-level discriminative information , which is beneficial for strengthening the discrimination ability of detectors . Concretely , we define a learned codebook C = { cj ∈ Rm , j = 1 , · · · , Q } that contains Q codewords . Each element di ∈ Rm of the map D can be assigned with a weight aij to each codeword cj and the corresponding residual vector is denoted by rij = di − cj , where i = 1 , 2 , · · · , n. Thus , dictionary learning can be calculated as follows : xj = n∑ i=1 aijrij , aij = exp ( −sj ||rij ||2 ) ∑Q j=1 exp ( −sj ||rij ||2 ) , ( 2 ) where sj indicates the learnable smoothing factor for the corresponding codeword cj . Finally , the output of dictionary learning is a fixed length representation X = { xj ∈ Rm , j = 1 , · · · , Q } . Next , we take E as the input of the RPN module to obtain a set of object proposals P ∈ Rz×m×o×o , where z and o separately denote the number of proposals and their spatial size . And the fusion result of P and X is taken as the input of the classifier . P = RPN ( E ) , y = cls ( [ φ ( P ) , wcX + bc ] ) , ( 3 ) where φ consists of two fully-connected layers . wc and bc are learnable parameters . ‘ [ , ] ’ indicates the fusion operation . Here , we use the concatenation operation . ‘ cls ’ denotes the classifier . By the constraint of the classification loss , we can promote the learned representation X and codebook C to absorb category-related information , which is good for enhancing detection accuracy . Dictionary-based Remap . To further facilitate the learned codebook C to retain more categoryrelated characteristics , we try to remap P to the dictionary space and perform classification . Concretely , each element p ∈ Rm of P is remapped as a combination of codewords in the codebook C. The processes are shown as follows : rep = Q∑ j=1 exp ( ψ ( p ) cTj ) ∑Q j=1 exp ( ψ ( p ) c T j ) cj , ( 4 ) where ψ is a fully-connected layer that maps p to the dictionary space . rep ∈ Rm indicates one element of the remapping output Rep ∈ Rz×m×o×o . Next , Rep is taken as the input of the classifier to output the probability : yrep = cls ( [ φ ( P ) , φ ( Rep ) ] ) , ( 5 ) where yrep indicates the output probability . Eq . ( 3 ) and Eq . ( 5 ) share the same classifier . Finally , the KL-Divergence loss Lkl is leveraged to enforce the prediction consistency between yrep and y . By performing classification in the dictionary space , the codebook C could be directly facilitated to learn category-related characteristics , which is conductive to the improvement of the discrimination ability .

Summary: The paper proposes a method for few shot object detection. It splits the feature map obtained from the backbone network into two parts. First is the component with largest valued singular values (similar to low rank matrix approximation with SVD), while other is that with the remaining lower ones. It then performs dictionary learning for coding the each column of the second matrix wrt the learned dictionary and uses the code/weight matrix as features. It fuses this with the earlier matrix and passes it to the region proposal network. Further it ‘remaps’ the object proposal features the same dictionary.