# A Novel One-Shot Object Detection via Multifeature Auxiliary Information

Authors: Yu Song; Min Li; Weidong Du; Yao Gou; Zhaoqing Wu; Yujie He
Journal: Security and Communication Networks
 (2022)

---

## Abstract

With the advantage of using only a limited number of samples, few-shot learning has been developed rapidly in recent years. It is mostly applied in the object classification or detection of a small number of samples which is typically less than ten. However, there is not much research related to few-shot detection, especially one-shot detection. In this paper, the multifeature information-assisted one-shot detection method is proposed to improve the accuracy of one-shot object detection. Specifically, two auxiliary modules are applied to the detection algorithm: Semantic Feature Module (SFM) and Detail Feature Module (DFM), which, respectively, extract semantic feature information and detailed feature information of samples in the support set. Then these two kinds of information are then calculated with the feature image extracted from the query image to obtain the corresponding auxiliary information that is used to complete one-shot detection. Thanks to the two auxiliary modules, which can retain more semantic and detailed information of samples in the support set, the proposed method can enhance the utilization rate of sample feature information and improve object detection accuracy by 2.97% compared to the benchmark method.


---

## Body

## 1. Introduction

Deep neural networks have been widely used in computer vision, such as posture recognition [1] and plant disease recognition [2], and object detection is the research hotspot in this field. Generally speaking, object detection algorithms can be divided into two categories according to different training strategies: one-stage object detection and two-stage object detection. The popular algorithms are the YOLO algorithms [3–5] and R-CNN algorithms [6–8], which dramatically improve the object detection effect and enhance detection efficiency. However, those algorithms rely on object annotation information, which cannot be easily obtained. Therefore, researchers gradually focus on few-shot detection.Few-shot detection is derived from few-shot learning, a particular case of meta-learning. At present, the learning methods can be roughly broken down into four categories: measurement learning-based learning, meta-learning-based learning, data enhancement-based learning, and multimodal approaches-based learning, among which the meta-learning-based learning method is the most popular. In the meta-training stage, by compositing several samples from different classes to take different meta-task, the model can learn the differences between examples of various categories and the similarities between samples of the same type. While in the meta-testing stage, the recognition task can be completed without retraining or only with a small amount of rapid training for a new category. However, few-shot learning is mainly used in few-shot classification rather than few-shot detection.Few-shot object detection is used to complete detection for objects with very few samples in the dataset. The existing few-shot detection methods fall into three categories: fine-tuning, model structure-based learning, and metric-based learning. The few-shot detection training strategy generally contains two stages: meta-training stage and fine-tuning stage. In the meta-training,N categories were randomly chosen from the training set, each containing K samples to form the support set of the model, namely, a meta-task. Next, a small number of object samples were selected to fine-tune the model. The purpose was to train the model to detect N classes of objects from N×K data and then generalize the knowledge to adapt to new classes. This task is called N-way K-shot. In few-shot learning, the K value is usually smaller than ten; when K=1, it is called one-shot learning.Existing metric-based few-shot detection mainly divides the dataset into the support set and the query set. It selects several image samples from the two sets to form the minimum training unit task (meta-task) and then trains the model through specific strategies. The detection algorithm first obtains the corresponding features of the images in the two sets, then measures the distance between the two features, and judges the object category according to the distance. According to the label’s location information, the regression operation is performed to complete the object positioning. We note that multiple features with different scales will be generated when the convolutional neural network extracts features of the support images. However, as the current algorithm only conducts simple distance measurement, the utilization rate of object feature information is extremely low.To solve this problem, this paper proposed a novel one-shot detection method on the basis of metric-based learning. The main contributions of this paper are as follows:(1)
This novel method integrated the Semantic Feature Module (SFM) and the Detail Feature Module (DFM), which generated features about the support images of two sizes (7×7 , 3×3). These features were then operated with query images’ feature and generated the corresponding multifeature auxiliary information (MFAI) of Semantic Feature Auxiliary Information and Detailed Feature Auxiliary Information.(2)
Experimental results showed that both the SFM and the DFM could increase the accuracy of one-shot detection. A combination of the two modules could even increase the detection accuracy by 2.97% compared to the original algorithm.


## 2. Related Works

In recent years, research on few-shot learning has attracted a lot of interest, which can effectively solve the classification and detection task using only a few labeled samples. The recent related works of few-shot classification, few-shot object detection, and one-shot object detection are listed in Table1.Table 1
Recent related works.
CategoryRef.MethodsFew-shot classification[9]The Mahalanobis distance in a state-of-the-art few-shot learning approach (CNAPS [10]) is adopted to improve performance[11]Presents a novel network to learn and preserve the feature manifold’s topology formed by different classes[12]Proposes the similarity ratio as an indicator for the generalization performance of a few-shot model[13]Takes advantage of the earth mover’s distance (EMD) to measure the distance between dense image representations which determines image relevance[14]Merges three learning methods: visual feature learning, knowledge inferring, and classifier learning, into a unified framework[15]Introduces the oPen sEt mEta LEaRning (PEELER), which randomly selects a set of novel classes, maximizes the posterior entropy over every sample, and utilizes the Mahalanobis distance as a new metricFew-shot object detection[16]Improves the CentreNet detector for the few-shot learning and a class-specific code generator is modeled by meta-learning[17]Uses Attention-RPN, multirelation detector and contrastive training strategy to detect novel objects[18]Proposes the model that uses labeled base categories and quickly improves to new categories, utilizing a meta-feature learner and a new upgraded moduleOne-shot object detection[19]Develops coattention and coexcitation framework (CoAE) that contributes to several technical aspects[20]Develops a new algorithm to guide the parameter posterior towards its true distribution to remedy the posterior fading problem that compromises the effectiveness of shared weightsIn the general one-shot object detection method, the weight extracted from the image is mainly used to measure the object feature distance. However, the semantic feature information and detailed feature information of the support set object are not fully utilized. This paper introduced the SFM and the Object Detail Module based on one-shot object detection, inspired by literature [18]. By using more object features to train the deep neural network, the detection effect of our model was better.


## 3. Method

The semantic information and detailed information need to be generated separately. In theory, the support images can obtain feature images of different sizes through different modules. In our method, the7×7 feature can retain more object details, while the 3×3 feature contains more semantic information about the object. The 7×7 feature and feature of the query image were used for dot product operation. The 3×3 feature is convolved with the feature of the query image. The corresponding MFAI of Semantic Feature Information and Detail Feature Information was generated through the above two operations.


### 3.1. Training Strategy

As mentioned above, in the training stage, assume that the dataset isD and divided into Dbase and Dnovel. Dbase represents an object image dataset which contains a large number of annotated images, of which category is Cbase; and Dnovel represents a dataset containing a few of samples with annotations with category as Cnovel. With Dbase∩Dnovel=∅, Cbase∩Cnovel=∅, so the ultimate goal of one-shot detection is to classify and locate the object of query image in Dnovel. Similar to the literature [16], the whole training process is divided into two steps. Firstly, the data in Cbase are used to train the model, so the model can learn the meta-features; then this trained model can have a good detection effect on the object of the base class. Lastly, the Dbase∪Dnovel dataset is utilized for model training and fine-tuning to adapt to the new category, then generalizing the knowledge learned in the first step to the new object category.Based on few-shot learning, we innovatively utilized the semantic information and detailed information of the object to complete the one-shot detection. Specifically, we used the smallest training unitT=Si,xii=1T in the training stage. The data in T are all from the randomly sampled support set in Dbase, Si, and the query image, xi. i represents the i−th task. Si contains N categories, with K samples in each class. Through multitask training, we obtained an object model with a detection base class. Next, the fine-tuned model was continuously adapted with the Dbase∪Dnovel dataset, and the detection model fθx|S was fine-tuned to fit the new category, in which θ is the parameter that the model needs to learn. The final one-shot detection could be completed by fθ tuned well.


### 3.2. Multifeature Information-Assisted Detection Method

Firstly, the feature extraction module was used to extract the7×7 feature image of query image. Then the support images were input into the DFM, SFM, and Weighted Module (WM), respectively, to get the corresponding 7×7, 3×3, 1×1 feature maps, while the channels were consistent. Next, we applied dot product 7×7 feature of support images with 7×7 feature generated by query image, and finally generated 7×7 feature—Detail Feature Auxiliary Information (DFAI), then the 7×7 feature generated by query image was convolved with the 3×3 feature of support images as a filter, and finally generated 7×7 feature—Semantic Feature Auxiliary Information (SFAI). The average operation was carried out on the two kinds of auxiliary information. The next step was to convolve the processed averaged feature (7×7 feature) with the weight information (1×1 feature) generated by the support images; thus, the 7×7 MFAI to be detected was generated. Finally, the detection feature map was put into the detection network to generate the classification and location information. The process was described as shown in Figure 1.Figure 1
The structure of the one-shot detection assisted by multiple feature information.To explain the function of each module in detail, we elaborated the function into two steps. The first step was to combine the Semantic Feature Auxiliary Module (SFAM) with the WM to output the feature to be detected; then combine the dot product information auxiliary module with the WM to output the feature to be detected. The former is illustrated in Figure2. Firstly, query image and support images were input into both the SFM and the feature extraction module to extract their respective features. Then the feature extraction module outputs the 1024×7×7 feature IQ, the SFM outputs the N×1024×3×3 feature SC, and the WM outputs the N×1024×1×1 feature SW. After the convolution operation of SC and IQ, FC was generated, as shown in (1)FC=SC∗IQ.Figure 2
Schema of the semantic feature auxiliary module.Then,SW and FC were convolved to get the N×1024×7×7 feature YC to be detected, as shown in(2)YC=SW∗FC,(3)YC=SW∗SC∗IQ.The DFM is illustrated in Figure3. Similarly, the query image and support images were put into the feature extraction module and the DFM to extract their respective features. So, the feature extraction module outputs the 1024×7×7 feature IQ, the DFM outputs the N×1024×7×7 feature SD, and the WM outputs the N×1024×1×1 feature SW. After the dot production operation of SD and IQ, FD was generated, as shown in(4)FD=SD⊗IQ.Figure 3
Schema of the detail feature auxiliary module.Then,SW and FD were convolved to obtain the N×1024×7×7 feature YD to be detected, as shown in(5)YD=SW∗FD,(6)YD=SW∗SD⊗IQ.


### 3.3. Loss Function

To handle the various objects which need to be detected in one-shot detection, the model in this paper adopted a softmax layer [18]. The predicted score on classification for the i-th class was represented by c^=eci/∑j=1Necj. To get better model convergence, the cross-entropy loss over the calibrated scores c^ was adopted, as shown in(7)Lc=−∑i=1NO⋅,ilogc^i,where O⋅,i is an indicator function. When the current anchor box fits into class i, its value is 1. Otherwise, the value is 0. For the bounding box regression calculation and object determination method, we followed the same detection way as YOLOV3. After anchors with different aspect ratios were preset, coordinate classification of anchors would be processed through calculation and finally predicted the object. In this paper, corresponding loss functions were adopted, such as the Mean Squared Error (MSE) loss and the Binary Cross-Entropy (BCE) loss. The loss function of multifeature information-assisted one-shot detection proposed in our model is shown in (8)Ldet=Lc+Lbbx+Lobj,where Lbbox is expressed as (9)Lbbox=λcoord∑i=0S2∑j=0BIijobj2−wi×hiai−a^i2+bi−b^i2+wi−w^i2+hi−h^i2,where S means the grid set in YOLO and S2 represents 13×13, 26×26, and 52×52. B stands for prediction box. Iijobj means the box at i, j, which is 1 if it is an object; otherwise, it is 0. ai, bi, wi, hi represent the central point coordinates and the width and height of the object, respectively. a^i, b^i, w^i, h^i represent the predicted values of the center point coordinates, width, and height, respectively.Lobj can be expressed as in(10)Lobj=λnoobj∑i=0S2∑j=0BIijnoobjci−c^i2+λobj∑i=0S2∑j=0BIijobjci−c^i2,where Iijnoobj means the box at i,j which is 1 if it is not an object; otherwise, it is 0. c^i is the prediction confidence on the object for class i. c^i is set to 1 if the object is the true value of a certain class; otherwise, it is 0.


## 3.1. Training Strategy

As mentioned above, in the training stage, assume that the dataset isD and divided into Dbase and Dnovel. Dbase represents an object image dataset which contains a large number of annotated images, of which category is Cbase; and Dnovel represents a dataset containing a few of samples with annotations with category as Cnovel. With Dbase∩Dnovel=∅, Cbase∩Cnovel=∅, so the ultimate goal of one-shot detection is to classify and locate the object of query image in Dnovel. Similar to the literature [16], the whole training process is divided into two steps. Firstly, the data in Cbase are used to train the model, so the model can learn the meta-features; then this trained model can have a good detection effect on the object of the base class. Lastly, the Dbase∪Dnovel dataset is utilized for model training and fine-tuning to adapt to the new category, then generalizing the knowledge learned in the first step to the new object category.Based on few-shot learning, we innovatively utilized the semantic information and detailed information of the object to complete the one-shot detection. Specifically, we used the smallest training unitT=Si,xii=1T in the training stage. The data in T are all from the randomly sampled support set in Dbase, Si, and the query image, xi. i represents the i−th task. Si contains N categories, with K samples in each class. Through multitask training, we obtained an object model with a detection base class. Next, the fine-tuned model was continuously adapted with the Dbase∪Dnovel dataset, and the detection model fθx|S was fine-tuned to fit the new category, in which θ is the parameter that the model needs to learn. The final one-shot detection could be completed by fθ tuned well.


## 3.2. Multifeature Information-Assisted Detection Method

Firstly, the feature extraction module was used to extract the7×7 feature image of query image. Then the support images were input into the DFM, SFM, and Weighted Module (WM), respectively, to get the corresponding 7×7, 3×3, 1×1 feature maps, while the channels were consistent. Next, we applied dot product 7×7 feature of support images with 7×7 feature generated by query image, and finally generated 7×7 feature—Detail Feature Auxiliary Information (DFAI), then the 7×7 feature generated by query image was convolved with the 3×3 feature of support images as a filter, and finally generated 7×7 feature—Semantic Feature Auxiliary Information (SFAI). The average operation was carried out on the two kinds of auxiliary information. The next step was to convolve the processed averaged feature (7×7 feature) with the weight information (1×1 feature) generated by the support images; thus, the 7×7 MFAI to be detected was generated. Finally, the detection feature map was put into the detection network to generate the classification and location information. The process was described as shown in Figure 1.Figure 1
The structure of the one-shot detection assisted by multiple feature information.To explain the function of each module in detail, we elaborated the function into two steps. The first step was to combine the Semantic Feature Auxiliary Module (SFAM) with the WM to output the feature to be detected; then combine the dot product information auxiliary module with the WM to output the feature to be detected. The former is illustrated in Figure2. Firstly, query image and support images were input into both the SFM and the feature extraction module to extract their respective features. Then the feature extraction module outputs the 1024×7×7 feature IQ, the SFM outputs the N×1024×3×3 feature SC, and the WM outputs the N×1024×1×1 feature SW. After the convolution operation of SC and IQ, FC was generated, as shown in (1)FC=SC∗IQ.Figure 2
Schema of the semantic feature auxiliary module.Then,SW and FC were convolved to get the N×1024×7×7 feature YC to be detected, as shown in(2)YC=SW∗FC,(3)YC=SW∗SC∗IQ.The DFM is illustrated in Figure3. Similarly, the query image and support images were put into the feature extraction module and the DFM to extract their respective features. So, the feature extraction module outputs the 1024×7×7 feature IQ, the DFM outputs the N×1024×7×7 feature SD, and the WM outputs the N×1024×1×1 feature SW. After the dot production operation of SD and IQ, FD was generated, as shown in(4)FD=SD⊗IQ.Figure 3
Schema of the detail feature auxiliary module.Then,SW and FD were convolved to obtain the N×1024×7×7 feature YD to be detected, as shown in(5)YD=SW∗FD,(6)YD=SW∗SD⊗IQ.


## 3.3. Loss Function

To handle the various objects which need to be detected in one-shot detection, the model in this paper adopted a softmax layer [18]. The predicted score on classification for the i-th class was represented by c^=eci/∑j=1Necj. To get better model convergence, the cross-entropy loss over the calibrated scores c^ was adopted, as shown in(7)Lc=−∑i=1NO⋅,ilogc^i,where O⋅,i is an indicator function. When the current anchor box fits into class i, its value is 1. Otherwise, the value is 0. For the bounding box regression calculation and object determination method, we followed the same detection way as YOLOV3. After anchors with different aspect ratios were preset, coordinate classification of anchors would be processed through calculation and finally predicted the object. In this paper, corresponding loss functions were adopted, such as the Mean Squared Error (MSE) loss and the Binary Cross-Entropy (BCE) loss. The loss function of multifeature information-assisted one-shot detection proposed in our model is shown in (8)Ldet=Lc+Lbbx+Lobj,where Lbbox is expressed as (9)Lbbox=λcoord∑i=0S2∑j=0BIijobj2−wi×hiai−a^i2+bi−b^i2+wi−w^i2+hi−h^i2,where S means the grid set in YOLO and S2 represents 13×13, 26×26, and 52×52. B stands for prediction box. Iijobj means the box at i, j, which is 1 if it is an object; otherwise, it is 0. ai, bi, wi, hi represent the central point coordinates and the width and height of the object, respectively. a^i, b^i, w^i, h^i represent the predicted values of the center point coordinates, width, and height, respectively.Lobj can be expressed as in(10)Lobj=λnoobj∑i=0S2∑j=0BIijnoobjci−c^i2+λobj∑i=0S2∑j=0BIijobjci−c^i2,where Iijnoobj means the box at i,j which is 1 if it is not an object; otherwise, it is 0. c^i is the prediction confidence on the object for class i. c^i is set to 1 if the object is the true value of a certain class; otherwise, it is 0.


## 4. Experiment

### 4.1. Experimental Environment

The running environment of the algorithm verification experiment is shown in Table2.Table 2
Runtime environment.
ItemParameterGPUTitan Xp (12G video memory)CUDA 10.0cuDNN 7.0Operating systemUbuntu 16.04Python version3.6Iterations60Learning rate0.001Momentum0.9Momentum attenuation coefficient0.00004


### 4.2. Dataset

The datasets we adopted are VOC 2007 [21] and VOC 2012 [22], which are the widely used object detection benchmarks. Out of the 20 categories, we selected samples of five categories as novel datasets and the remaining 15 categories as the base datasets. The model training was divided into two stages: base training stage and one-shot fine-tuning stage. In the base training stage, the images of base samples were used to train the normal model in the supervised mode. And in the one-shot fine-tuning stage, the images of novel ones were used to ensure that each class of objects only had one annotated bounding box.


### 4.3. Experiment and Analysis

A large number of experiments have been done on these datasets. To illustrate the effectiveness of the different methods, several representative datasets were selected. In the test phase, we used five unseen categories of data in training: bird, bus, cow, motorbike, and sofa. Due to the space limitation, we only present the results data of bird and bus categories in Figures4 and 5.Figure 4
Bird object detection results.Figure 5
Bus object detection results.As can be seen from the above Figure4, there are a total of 12 images arranged in four rows, with three object images in each row. All the object images used the same kind of detection algorithm. The first line to the fourth line, respectively, shows the detection results of the original algorithm [18], the detailed feature information auxiliary algorithm, the semantic feature information auxiliary algorithm, and the multifeature information auxiliary algorithm proposed in this paper. Different detection results of the same object are presented in each column for those four algorithms. For column (a), the object is conspicuous, so all four algorithms can correctly identify it. As the background of the object in column (b) is relatively complex, the original algorithm and the auxiliary algorithm of detailed feature information cannot detect the object well. In contrast, the algorithms in the third and fourth lines that integrate semantic information can detect the object more accurately. Since the objects in column (c) belong to multiobject detection in a complex background; the detection effect of the second row is not ideal. The algorithms in the first and third rows can completely detect conspicuous objects. While the fourth line algorithm can detect multiple objects, the second object detection is not complete because there is little difference between the background color and the object color.Similarly, there are also 12 images in Figure5, which are arranged in the same manner. The object image of each row uses the same kind of detection algorithm. The first row to the fourth row, respectively, represents the detection results of different buses by the four algorithms. For column (a), the object is prominent, so all the four algorithms can correctly identify it. In column (b), the background of images is relatively complex. Although the original algorithm and the detailed feature assist algorithm can locate the object more accurately, there is still a misjudgment in the classification of extracted features. As a result, the bus is misclassified as a train. The algorithms in the third and fourth rows incorporate semantic information to detect objects more accurately. Since the object in column (c) is similar to a train, the four algorithms misjudge the result. That is why the accuracy of bus detection results is low.


### 4.4. Ablation Experiments

The accuracy improvement of our proposed method was due to the multifeature auxiliary detection mechanisms, that is, SFM and DFM. To illustrate the importance of these modules, we implemented ablation experiments by disabling different modules.


#### 4.4.1. Semantic Feature Auxiliary Detection Algorithm

Figure6 shows the flow chart of the semantic feature auxiliary detection algorithm. Two modules were constructed, namely, the SFAM and the WM. The semantic feature-assisted detection algorithm was compared with the original algorithm in the experiment. As can be seen in Table 3, semantic feature assistance achieves better performance.Figure 6
One-shot detection algorithm structure assisted by semantic information.Table 3
Ablation experiment results.
MethodsNovelBirdBusCowMbikeSofamAPBenchmark algorithm [16]24.123.4524.9726.4427.5121.3Detail feature auxiliary detection24.523.2726.5827.0328.1421.91Semantic feature auxiliary detection25.376.0326.3228.0929.4423.05Multifeature auxiliary detection27.028.5127.828.6729.3524.27The bold values indicate that the performance of the proposed method is better than that of the Benchmark algorithm.


#### 4.4.2. Detail Feature Auxiliary Detection Algorithm

The detail feature auxiliary detection algorithm was implemented, as shown in Figure7. Two modules were also constructed: the detail feature auxiliary module (DFAM) and the WM. In the experiment, this detail feature-assisted detection algorithm was also compared with the original algorithm. As indicated in Table 3, the performance of the detail feature-assisted detection algorithm is only better than that of the basic weight network.Figure 7
One-shot detection algorithm structure assisted by detailed information.


#### 4.4.3. Multifeature Auxiliary Detection Algorithm

In the multifeature auxiliary detection algorithm, both the above modules were introduced into the network and fused with the basic weighted network structure.The detection results of the above three algorithms and the original benchmark algorithm are shown in Table3.Table3 shows the comparison results of those algorithms proposed in this paper and the original algorithm, from which we can see the performance of the designed modules. On the left side of the table are different algorithms, while on the right side are the corresponding experimental results. The average precision (AP) of objects in the five classifications has also been calculated separately for the bird, bus, cow, motorbike, and sofa. The mAP represents the mean of AP for these five novel classes. The first row is the original algorithm, and the other three are related algorithms proposed in this paper. The second and third rows are the results of ablation experiments. The second algorithm adds the DFAM to the original algorithm, which is called the detail feature auxiliary detection algorithm. It can be observed that the AP for the bird, cow, motorbike, and sofa is higher than that of the original algorithm, while it is not the case for the bus. Well, the performance is improved by less than 1%. The third row adds the SFAM to the original algorithm, which is called the semantic feature auxiliary detection algorithm. It can be observed that the AP of the algorithm is higher than that of the original algorithm, with an increase of 2.58% in AP and 1.75% in mAP for the bus. It can also be seen that the SFAM proposed in this paper does enhance the semantic information of objects and promotes classification accuracy and detection precision. The fourth row is the result of the multifeature auxiliary detection algorithm proposed in this paper. Notably, the algorithm has enhanced the detailed information and semantic information, and the detection results of all those five novel objects are superior to the original algorithm. In particular, the AP of the bus has been improved by 5.06%, and the mAP has been improved by 2.97%. The detection results in the fourth row show that the detail feature and semantic feature, two auxiliary information introduced in the algorithm, successfully improve the AP and mAP of one-shot object detection.


## 4.1. Experimental Environment

The running environment of the algorithm verification experiment is shown in Table2.Table 2
Runtime environment.
ItemParameterGPUTitan Xp (12G video memory)CUDA 10.0cuDNN 7.0Operating systemUbuntu 16.04Python version3.6Iterations60Learning rate0.001Momentum0.9Momentum attenuation coefficient0.00004


## 4.2. Dataset

The datasets we adopted are VOC 2007 [21] and VOC 2012 [22], which are the widely used object detection benchmarks. Out of the 20 categories, we selected samples of five categories as novel datasets and the remaining 15 categories as the base datasets. The model training was divided into two stages: base training stage and one-shot fine-tuning stage. In the base training stage, the images of base samples were used to train the normal model in the supervised mode. And in the one-shot fine-tuning stage, the images of novel ones were used to ensure that each class of objects only had one annotated bounding box.


## 4.3. Experiment and Analysis

A large number of experiments have been done on these datasets. To illustrate the effectiveness of the different methods, several representative datasets were selected. In the test phase, we used five unseen categories of data in training: bird, bus, cow, motorbike, and sofa. Due to the space limitation, we only present the results data of bird and bus categories in Figures4 and 5.Figure 4
Bird object detection results.Figure 5
Bus object detection results.As can be seen from the above Figure4, there are a total of 12 images arranged in four rows, with three object images in each row. All the object images used the same kind of detection algorithm. The first line to the fourth line, respectively, shows the detection results of the original algorithm [18], the detailed feature information auxiliary algorithm, the semantic feature information auxiliary algorithm, and the multifeature information auxiliary algorithm proposed in this paper. Different detection results of the same object are presented in each column for those four algorithms. For column (a), the object is conspicuous, so all four algorithms can correctly identify it. As the background of the object in column (b) is relatively complex, the original algorithm and the auxiliary algorithm of detailed feature information cannot detect the object well. In contrast, the algorithms in the third and fourth lines that integrate semantic information can detect the object more accurately. Since the objects in column (c) belong to multiobject detection in a complex background; the detection effect of the second row is not ideal. The algorithms in the first and third rows can completely detect conspicuous objects. While the fourth line algorithm can detect multiple objects, the second object detection is not complete because there is little difference between the background color and the object color.Similarly, there are also 12 images in Figure5, which are arranged in the same manner. The object image of each row uses the same kind of detection algorithm. The first row to the fourth row, respectively, represents the detection results of different buses by the four algorithms. For column (a), the object is prominent, so all the four algorithms can correctly identify it. In column (b), the background of images is relatively complex. Although the original algorithm and the detailed feature assist algorithm can locate the object more accurately, there is still a misjudgment in the classification of extracted features. As a result, the bus is misclassified as a train. The algorithms in the third and fourth rows incorporate semantic information to detect objects more accurately. Since the object in column (c) is similar to a train, the four algorithms misjudge the result. That is why the accuracy of bus detection results is low.


## 4.4. Ablation Experiments

The accuracy improvement of our proposed method was due to the multifeature auxiliary detection mechanisms, that is, SFM and DFM. To illustrate the importance of these modules, we implemented ablation experiments by disabling different modules.


### 4.4.1. Semantic Feature Auxiliary Detection Algorithm

Figure6 shows the flow chart of the semantic feature auxiliary detection algorithm. Two modules were constructed, namely, the SFAM and the WM. The semantic feature-assisted detection algorithm was compared with the original algorithm in the experiment. As can be seen in Table 3, semantic feature assistance achieves better performance.Figure 6
One-shot detection algorithm structure assisted by semantic information.Table 3
Ablation experiment results.
MethodsNovelBirdBusCowMbikeSofamAPBenchmark algorithm [16]24.123.4524.9726.4427.5121.3Detail feature auxiliary detection24.523.2726.5827.0328.1421.91Semantic feature auxiliary detection25.376.0326.3228.0929.4423.05Multifeature auxiliary detection27.028.5127.828.6729.3524.27The bold values indicate that the performance of the proposed method is better than that of the Benchmark algorithm.


### 4.4.2. Detail Feature Auxiliary Detection Algorithm

The detail feature auxiliary detection algorithm was implemented, as shown in Figure7. Two modules were also constructed: the detail feature auxiliary module (DFAM) and the WM. In the experiment, this detail feature-assisted detection algorithm was also compared with the original algorithm. As indicated in Table 3, the performance of the detail feature-assisted detection algorithm is only better than that of the basic weight network.Figure 7
One-shot detection algorithm structure assisted by detailed information.


### 4.4.3. Multifeature Auxiliary Detection Algorithm

In the multifeature auxiliary detection algorithm, both the above modules were introduced into the network and fused with the basic weighted network structure.The detection results of the above three algorithms and the original benchmark algorithm are shown in Table3.Table3 shows the comparison results of those algorithms proposed in this paper and the original algorithm, from which we can see the performance of the designed modules. On the left side of the table are different algorithms, while on the right side are the corresponding experimental results. The average precision (AP) of objects in the five classifications has also been calculated separately for the bird, bus, cow, motorbike, and sofa. The mAP represents the mean of AP for these five novel classes. The first row is the original algorithm, and the other three are related algorithms proposed in this paper. The second and third rows are the results of ablation experiments. The second algorithm adds the DFAM to the original algorithm, which is called the detail feature auxiliary detection algorithm. It can be observed that the AP for the bird, cow, motorbike, and sofa is higher than that of the original algorithm, while it is not the case for the bus. Well, the performance is improved by less than 1%. The third row adds the SFAM to the original algorithm, which is called the semantic feature auxiliary detection algorithm. It can be observed that the AP of the algorithm is higher than that of the original algorithm, with an increase of 2.58% in AP and 1.75% in mAP for the bus. It can also be seen that the SFAM proposed in this paper does enhance the semantic information of objects and promotes classification accuracy and detection precision. The fourth row is the result of the multifeature auxiliary detection algorithm proposed in this paper. Notably, the algorithm has enhanced the detailed information and semantic information, and the detection results of all those five novel objects are superior to the original algorithm. In particular, the AP of the bus has been improved by 5.06%, and the mAP has been improved by 2.97%. The detection results in the fourth row show that the detail feature and semantic feature, two auxiliary information introduced in the algorithm, successfully improve the AP and mAP of one-shot object detection.


## 4.4.1. Semantic Feature Auxiliary Detection Algorithm

Figure6 shows the flow chart of the semantic feature auxiliary detection algorithm. Two modules were constructed, namely, the SFAM and the WM. The semantic feature-assisted detection algorithm was compared with the original algorithm in the experiment. As can be seen in Table 3, semantic feature assistance achieves better performance.Figure 6
One-shot detection algorithm structure assisted by semantic information.Table 3
Ablation experiment results.
MethodsNovelBirdBusCowMbikeSofamAPBenchmark algorithm [16]24.123.4524.9726.4427.5121.3Detail feature auxiliary detection24.523.2726.5827.0328.1421.91Semantic feature auxiliary detection25.376.0326.3228.0929.4423.05Multifeature auxiliary detection27.028.5127.828.6729.3524.27The bold values indicate that the performance of the proposed method is better than that of the Benchmark algorithm.


## 4.4.2. Detail Feature Auxiliary Detection Algorithm

The detail feature auxiliary detection algorithm was implemented, as shown in Figure7. Two modules were also constructed: the detail feature auxiliary module (DFAM) and the WM. In the experiment, this detail feature-assisted detection algorithm was also compared with the original algorithm. As indicated in Table 3, the performance of the detail feature-assisted detection algorithm is only better than that of the basic weight network.Figure 7
One-shot detection algorithm structure assisted by detailed information.


## 4.4.3. Multifeature Auxiliary Detection Algorithm

In the multifeature auxiliary detection algorithm, both the above modules were introduced into the network and fused with the basic weighted network structure.The detection results of the above three algorithms and the original benchmark algorithm are shown in Table3.Table3 shows the comparison results of those algorithms proposed in this paper and the original algorithm, from which we can see the performance of the designed modules. On the left side of the table are different algorithms, while on the right side are the corresponding experimental results. The average precision (AP) of objects in the five classifications has also been calculated separately for the bird, bus, cow, motorbike, and sofa. The mAP represents the mean of AP for these five novel classes. The first row is the original algorithm, and the other three are related algorithms proposed in this paper. The second and third rows are the results of ablation experiments. The second algorithm adds the DFAM to the original algorithm, which is called the detail feature auxiliary detection algorithm. It can be observed that the AP for the bird, cow, motorbike, and sofa is higher than that of the original algorithm, while it is not the case for the bus. Well, the performance is improved by less than 1%. The third row adds the SFAM to the original algorithm, which is called the semantic feature auxiliary detection algorithm. It can be observed that the AP of the algorithm is higher than that of the original algorithm, with an increase of 2.58% in AP and 1.75% in mAP for the bus. It can also be seen that the SFAM proposed in this paper does enhance the semantic information of objects and promotes classification accuracy and detection precision. The fourth row is the result of the multifeature auxiliary detection algorithm proposed in this paper. Notably, the algorithm has enhanced the detailed information and semantic information, and the detection results of all those five novel objects are superior to the original algorithm. In particular, the AP of the bus has been improved by 5.06%, and the mAP has been improved by 2.97%. The detection results in the fourth row show that the detail feature and semantic feature, two auxiliary information introduced in the algorithm, successfully improve the AP and mAP of one-shot object detection.


## 5. Conclusion

In this paper, a novel one-shot detection method based on multifeature auxiliary information was proposed. Compared to previous studies, this algorithm utilized two auxiliary mechanisms: the Semantic Feature Module and the Detail Feature Module, which significantly improved the detection effect of a single sample object. Experimental results on public datasets demonstrate that the new proposed algorithm has better one-shot detection performance than the original method. To further evaluate the advanced performance, an ablation experiment was conducted. Experiments showed that the two auxiliary modules play a positive role in the detection results. The combined detection accuracy of the two modules has been increased by 2.97% compared to the benchmark algorithm. In the future, we will apply this proposed method to other types of datasets, including infrared images and SAR images. Although this method is mainly for one-shot object detection, we also look forward to its application in few-shot object detection.



---

*Source: 6805526-2022-02-21.xml*