Year: 2021

Abstract:
Tiny objects are hard to detect due to their low resolution and small size. The poor detection performance of tiny objects is mainly caused by the limitation of network and the imbalance of training dataset. A new feature pyramid network is proposed to combine context augmentation and feature refinement. The features from multi-scale dilated convolution are fused and injected into feature pyramid network from top to bottom to supplement context information. The channel and spatial feature refinement mechanism is introduced to suppress the conflicting formation in multi-scale feature fusion and prevent tiny objects from being submerged in the conflict information. In addition, a data enhancement method called copy-reduce-paste is proposed, which can increase the contribution of tiny objects to loss during training, ensuring a more balanced training. Experimental results show that the mean average precision of target targets on the VOC dataset of the proposed network reaches 16.9% (IOU=0.5:0.95), which is 3.9% higher than YOLOV4, 7.7% higher than CenterNet, and 5.3% higher than RefineDet.

text: Tiny objects are hard to detect due to their low resolution and small size. The poor detection performance of tiny objects is mainly caused by the limitation of network and the imbalance of training dataset. A new feature pyramid network is proposed to combine context augmentation and feature refinement. The features from multi-scale dilated convolution are fused and injected into feature pyramid network from top to bottom to supplement context information. The channel and spatial feature refinement mechanism is introduced to suppress the conflicting formation in multi-scale feature fusion and prevent tiny objects from being submerged in the conflict information. In addition, a data enhancement method called copy-reduce-paste is proposed, which can increase the contribution of tiny objects to loss during training, ensuring a more balanced training. Experimental results show that the mean average precision of target targets on the VOC dataset of the proposed network reaches 16.9% (IOU=0.5:0.95), which is 3.9% higher than YOLOV4, 7.7% higher than CenterNet, and 5.3% higher than RefineDet.

heading: 1 INTRODUCTION
text: As an challenge in the field of target detection, tiny object detection is widely used in vision tasks such as autonomous driving, medical field, drone navigation, satellite positioning, and industrial detection. In recent years, object detectors based on deep learning have made great progress(Tong et al., 2020; Feng et al., 2020). One-stage algorithms represented by(Redmon & Farhadi, 2018; Zhang et al., 2020; Liu et al., 2016) can directly get access to the target of interest through the forward convolutional neural network with a fast speed. However, two-stage algorithms(He et al., 2017; Ren et al., 2016) obtain the RoI (Region of Interest) based on the generated candidate boxes, which has higher accuracy. Although these target detection algorithms have made great progress in precision and speed, their performance is still very unsatisfactory when detecting tiny targets(less than 32×32 pixels). On most public data sets, the detection precision of tiny objects is even less than half of that of larger targets(Liu et al., 2016). Therefore, tiny object detection still has a lot of room for improvement.
The poor performance of tiny object detection is mainly caused by the limitations of the network itself and the imbalance of training data(Kisantal et al., 2019). To obtain solid semantic information, modern detectors try to stack more and more pooling and down-sampling operations so that tiny object features with few pixels are gradually lost in forwarding propagation(Liu et al., 2021), limiting the detection performance of tiny objects. FPN(Lin et al., 2017) can alleviate the problem of information diffusion to a certain extent (Redmon & Farhadi, 2018; Liu et al., 2016) by fusing low-resolution feature maps with high-resolution feature maps horizontally. However, fusing the information of different densities directly will cause semantic conflicts, which limiting the expression of multi-scale features and making tiny objects submerged in conflicting information easily. At the same time, in the current classic public data set, the number of annotations of tiny objects is much less than that of larger targets(Chen et al., 2020). Therefore, the convergence direction of the network is continuously leaning toward larger targets during training, resulting in poor performance of tiny objects. Consequently, we believe that it is feasible to improve the detection rate of tiny objects from the above two aspects.
To solve the problem of feature dispersion of tiny objects and semantic differences between layers, this paper proposes a new feature pyramid composite neural network structure that combines context augmentation and feature refinement. The proposed algorithm framework is shown in Figure 1.
It is combined with context augmentation module (CAM) and feature refinement module (FRM). CAM merges multi-scale dilated convolution features to obtain rich context information for feature augmentation. FRM introduces a feature refinement mechanism in the dimensions of the channel and space to suppress conflict information and prevent tiny objects from being submerged in conflict information. Simultaneously, to ensure that the network will not lean towards larger targets during training, a method called copy-reduce-paste is proposed to increase the contribution to the loss of tiny objects in training. We train and test on the standard public data set PASCAL VOC, and verify the effectiveness of out method for detecting tiny objects through comparative experiments and ablation experiments. The algorithm proposed in this paper achieves a precision of 83.6% (IOU=0.5) on the VOC data set, which is higher than most comparison algorithms, and the precision of tiny objects reaches 16.9% (IOU=0.5:0.95), which is higher than YOLOV4, CenterNet and other cutting-edge networks.

heading: 2 RELATED WORK

heading: 2.1 OBJECT DETECTOR BASED ON DEEP LEARNING
text: As a fundamental computer vision task, target detection contains both classification and localization, which can be regarded as a regression problem. In the early days, hand-designed features were widely applied to target detection. However, hand-designed feature is a kind of shallow feature, and it is gradually replaced after the appearance of the CNN-based features. R-CNN(He et al., 2017), as the pioneering work of two-stage algorithms, employs prior boxes of different size to match targets of different size and then selects candidate regions through CNN. To reduce the training time, FastRCNN(Ren et al., 2016; Xiao et al., 2020) extracts the feature map of the entire image, and then spatial pyramid pooling and RoI (Region of Interest) pooling are used to generate regional features and to filter candidate regions. To further improve the precision of tiny object, EFPN(Deng et al., 2021) proposes a super-resolution feature pyramid structure to amplify tiny object features. Compared with two-stage networks, one-stage networks have a faster speed but lower precision. SSD(Liu et al., 2016) puts anchor boxes densely on the image to obtain the target boxes, and meanwhile, it makes full use of features of different scales to detect smaller targets. YOLOV3(Redmon & Farhadi, 2018) chooses to detect large, medium, and tiny objects separately based on three outputs of the feature pyramid, which significantly improves the detection performance of tiny objects.(Zhang et al., 2020) introduces a high-resolution attention mechanism to FPN to mine the most useful information of tiny targets. This paper chooses YOLOV3(Redmon & Farhadi, 2018) as the baseline and makes
improvements on this basis. RefineDet(Zhang et al., 2018) introduces a new loss function to solve the imbalance between simple samples and difficult samples. Recently, detectors based on anchorfree architecture are becoming more and more popular(Zhao et al., 2019). Although target detection algorithms are constantly developing and replacing, there is no big breakthrough in the field of tiny object detection, and the detection precision of tiny objects stays low.

heading: 2.2 MULTI-SCALE FEATURE FUSION
text: Using multi-scale features is an effective method to improve the detection precision of tiny objects. SSD(Liu et al., 2016) is the first attempt to predict the location and category of targets with multiscale features. FPN(Lin et al., 2017) merges adjacent feature maps with different grains from top to bottom, which can improve the expressive ability of features greatly. A large number of variant structures similar to FPN(Lin et al., 2017) have emerged. PANet(Liu et al., 2018) adds extra bottomup connections based on FPN(Lin et al., 2017) to transfer information from the lower layer to the upper layer more efficiently. NAS-FPN(Ghiasi et al., 2019) found a new connection method through neural architecture search technology. BiFPN(Tan et al., 2020) improved the connection method of PANet(Liu et al., 2018) to make it more efficient and introduced a simple attention mechanism at the connection point. Although the structures mentioned above have greatly improved the multi-scale expression ability of the network, they have ignored the existence of conflict information between features of different scales, and the lack of context information may hinder the further improvement of performance, especially for tiny objects, which is easy to be submerged in conflict information. This article fully considers the impact of conflict information and context information on detection precision.

heading: 2.3 DATA AUGMENTATION
text: Preprocessing of the training set has always been an indispensable part of deep learning, such as rotation, deformation, random erasure, random occlusion, illumination distortion, and MixUp. In recent days, several data enhancement methods for tiny objects have been proposed.(Chen et al., 2020) regards loss as a kind of feedback. And four images were scaled to the same size and stitched together to enhance the performance of tiny object detection under the guidance of feedback. Unlike(Chen et al., 2020), (Yu & Koltun, 2015) scales 4 images to different sizes and stitched them into one.(Kisantal et al., 2019) tried to achieve tiny object data augmentation by copying and pasting tiny objects back to original images. This method can only increase the number of tiny objects but not the number of training images containing tiny objects. It will also cause the imbalance of training to a certain extent. Because larger targets are widely distributed in each batch of training, this paper guarantees the tiny objects’ contribution to the loss in eache batch of training, making the training more balanced.

heading: 3 PROPOSED METHODS
text: This chapter will introduce our tiny object detection network in detail. As we can see in Figure 1, {C2, C3, C4, C5} represent different levels after input image being down-sampled by {4, 8, 16, 32} times. {F1, F2, F3} are denoted as newly generated feature levels corresponding to {C3, C4, C5} by a layer of convolution, and C2 is discarded because of a mess of noises. {L1, L2, L3} are denoted as feature levels generated by FPN and {P1, P2, P3} are denoted as the feature levels generated by FRM. The network is mainly composed of CAM and FRM. CAM is inspired by the mode that humans recognize objects. For example, it is difficult for human to distinguish a bird in a very high sky, but it is easy for human to distinguish when considering the sky as the context information. Therefore, we believe that context information is helpful for tiny object detection. CAM applies dilated convolution with different dilated convolution rates to obtain context information of different receptive fields, and injects it into FPN(Lin et al., 2017) from top to bottom to enrich context information. But it will introduce redundant information and conflicting information while sharing the information, because of the semantic differences among different levels of FPN(Lin et al., 2017). Therefore, FRM is proposed to filter conflict information and reduce semantic differences. By fusing the features between different layers adaptively, the conflict information among layers is eliminated to prevent the tiny object features from being submerged in the conflict information.
Simultaneously, in view of the small number of positive samples generated by tiny objects and the limited contribution to loss of tiny objects, a data augmentation method called copy-reduce-paste is proposed. Specifically, copy the larger targets in the training set, reduce them, and then paste them back to the original imge. During the pasting process, it is necessary to ensure that the pasted targets do not overlap with the existing targets. The above methods will be explained in detail in the following sections.

heading: 3.1 FEATURE PYRAMID NETWORK WITH CONTEXT AUGMENTATION AND FEATURE REFINEMENT

heading: 3.1.1 CONTEXT AUGMENTATION MODULE
text: Tiny target detection requires context information. We propose to use dilated convolution with different dilated convolution rates to obtain context information of different receptive fields to enrich the context information of FPN. The structure is shown in Figure 2.
Figure 2 is the structure of CAM(Yu & Koltun, 2015). We obtain the context information of different receptive fields by performing dilated convolution with different dilated convolution rates on C5. The kernel size is 3×3, and the dilated convolution rates are 1, 3, and 5. The possible ways of fusion are shown in Figure 3 (a), (b), and (c).
Method (a) and (c) are weighted fusion and concatenation operation respectively. That is, the feature maps are directly added in the dimension of space and channel. Method (b) is an adaptive fusion method. Specifically, assuming that the size of the input can be denoted as (bs, C, H, W), we can obtain a spatial adaptive weight of (bs, 3, H, W) by performing operations of convolution,
concatenation and Softmax. Three channels correspond to the three inputs one-to-one, and the context information can be aggregated to the output by calculating the weighted sum. We verify the effectiveness of each fusion method through ablation experiments and the results are shown in the following Table 1. APs, APm, and APl are defined as the precision of tiny, medium, and large targets. And ARs, ARm, and ARl are denoted as the recall of tiny, medium, and large targets.
It can be seen from Table 1 that the advantages obtained by (c) is the largest for tiny objects. APs and ARs are both increased by 1.8%. Method (b) has the greatest improvement for medium and large targets. The improvement brought by method (a) is basically somewhere in between.

heading: 3.1.2 FEATURE REFINEMENT MODULE
text: FPN(Lin et al., 2017) is proposed to fuse features of different scales. However, features of different scales have semantic differences that cannot be ignored. Directly fusing features of different scales will bring much redundant information and conflicting information, reducing the ability of multiscale expression. Therefore, FRM is proposed to filter conflicting information and prevent tiny object features from being submerged in conflict information. The overall structure of FRM is shown in Figure 4.
As can be seen from Figure 4, FRM is mainly composed of two parallel branches, namely channel purification module and space purification module. They are used to generate adaptive weights in the dimension of space and channel, which can guide features to learn in a more critical direction. The structure of channel purification module is shown in Figure 4(b). To obtain channel attention map, the input feature map is compressed in the dimension of space to aggregate the spatial information that can represent the global features of images. Adaptive average pooling and adaptive maximum pooling are combined to obtain more refined global features of images. Xm is defined as the input
of the mth (m={1, 2, 3}) layer of FRM. X(n,m) is defined as the result of resizing from the nth layer to the mth layer. Xmk,x,y are defined as the value of the mth feature map on the kth channel at the position (x,y). So, the output of the upper branch is:
K m x,y = a m ·X(1,m)x,y + bm ·X(2,m)x,y + bm ·X(3,m)x,y (1)
In the above formula, Kmx,y represents the output vector of the mth layer at the position (x,y). a, b and c are the channel adaptive weights, the size of which is 1×1×1. a,b, and c are defined as:
[am, b m, cm] = σ [AP (F ) +MP (F )] (2)
F is the feature generated by concatenation operation just as show in Figure 4. σ represents the sigmoid operation. AP and MP are average pooling and maximum pooling respectively, and then these two weights are summed in the dimension of space, and the channel-based adaptive weight is generated after sigmoid.
The spatial purification module generates the relative weight of all positions relative to the channel through softmax, and the output of the lower branch is shown in the following equation 3:
φmx,y= 3∑ c=1 ∑ k,x,y ( µmc,x,y·X (1,m) k,x,y +ν m c,x,y·X (2,m) k,x,y +η m c,x,y·X (3,m) k,x,y ) (3)
In formula 3, x and y denote the spatial position of the feature map, and k denotes the channel of the input feature map. φmx,y is the output feature vector at position (x,y). µ m c,x,y ,ν m c,x,y and η m c,x,y denote the spatial attention weight relative to the mth layer, where c represents their channel. µ,ν,η can be expressed by the formula 4:
[µm, νm, ηm] = Softmax (F ) (4)
In formula 4, F has the same meaning as formula 2, and softmax is used to normalize the feature map in the direction of the channel to get the relative weights of different channels at the same location. Therefore, the total output of this module can be expressed as:
pm = φm +K m (5)
In this way, the features of all layers of FPN are fused together under the guidance of adaptive weights, and {p1, p2, p3} is used as the final output of the entire network. In this way, the features of all layers of FPN are fused together under the guidance of adaptive weights, and {p1, p2, p3} is used as the final output of the entire network. To demonstrate the effectiveness of FRM, we visualized some feature maps. The detection of tiny objects is mainly dominated by the bottom layer of FPN, so the bottom layer features are visualized in this section only. And we scale the feature maps to the same size. As shown in the figure, the leftmost column is the input image to be detected. F3, L3, P3 are the visualization results of the feature map of the corresponding label in Figure 1.
It can be seen from Figure 5 that F3 can roughly locate the position of targets, but there is more noise in the background. After FPN(Lin et al., 2017), a large amount of high-level semantic information is introduced into L3. These features can filter most of the background noise, but conflict information is also introduced because of the different grains of feature, making the response of the target area weaken. Focusing on P3, the target feature is strengthened, the background area is suppressed, and the boundary between the target and the background is more obvious, which will help the detector distinguish between positive and negative samples and facilitate positioning and classification. It can be seen from the visual analysis that the FRM proposed in this paper can greatly reduce the conflict information and improve the detection precision of tiny objects.

heading: 3.2 COPY-REDUCE-PASTE DATA ENHANCEMENT
text: In the current mainstream public data set, the number of positive samples generated by tiny objects and the contribution to loss of tiny objects is much smaller than those of larger targets, making the direction of convergence lean toward to larger targets. In order to alleviate this problem, we copy, reduce, and paste the target in images during training. By increasing the number of tiny objects in images and the number of images containing tiny objects, the contribution to the loss of the tiny object is increased and makes training more balanced. Figure 6 b, c is the results of pasting once per target at different positions. By this way, the number and context information of tiny objects is greatly enriched.
In this part, we study the influence of the paste times on the detection of tiny objects. The results of the ablation experiment are shown in Table 2.
It can be seen from Table 2 that as the times of pasting increase, the detection performance of tiny objects gradually decreases, and it may even be lower than the baseline. This may because that as the number of pasting increases, the distribution of the data set is gradually destroyed, making the performance in the test set worse. Experimental results show that pasting once is the best setting. Compared with the baseline, APs is increased by 2.5%, ARs is increased by 1.9%, and the detection performance of medium and large targets is also slightly improved.

heading: 3.3 ABLATION STUDY
text: We design ablation experiments to verify the effectiveness and contribution rate of each module. In this section, we gradually add data enhancement methods, CAM, and FRM to the baseline model, YOLOV3(Redmon & Farhadi, 2018), ensuring that the test environment and configuration are exactly the same during the experiment. The experimental results are shown in Table 3.
In general, the module proposed in this paper can significantly improve the target detection performance, especially for the tiny objects and medium objects, which is also in line with our original intention. As shown in the table, APs is increased by 5.4%. APm is increased by 3.6% , while APl is increased by 1.0%. At the same time, the recall of targets of different scales has also been improved to varying degrees. Specifically, ARs is increased by 6.9%, ARm is increased by 2.3%, and ARl is increased by 1.1%. Copy-reduce-paste: The data enhancement method increases APs by 2.5% , increases APm by 2.2%, but decreases APl slightly. CAM: The CAM module can improve APs , APm, and APl, especially for APs . Its precision and recall rate are increased by 1.8% and 1.9% respectively. FRM: APs is increased by 2.8%, APm is increased by 1.6%, and APl is basically the same.

heading: 4 EXPERIMENT

heading: 4.1 DATA SET AND EVALUATION INDICATORS
text: Experiments are conducted on PASCAL VOC data. The data set has 20 classes and contains 22136 training images (voc2007 and 2012trianval) and 4952 test images (voc2007test). we choose average precision (AP) and average recall (AR) as evaluation indicators. The precision evaluation indicators contain mAP, APs, and APm, which measure the overall precision rate, tiny object precision rate, and medium target precision rate, respectively. The recall evaluation indicators contain AR, ARs, and ARm, which measure the overall recall rate, the tiny object recall rate, and the medium target recall rate, respectively.
In order to ensure the fairness of comparison, all experiments in this paper are conducted under the framework of PyTorch(Paszke et al., 2019), and hardware facilities are kept the same(CPU: Intel Core i7-5820k CPU@3.30GHZ, Memory: 16GB, Graphics card: GeForce GTX TITAN X). We apply the SGD optimizer to train 50 epochs, set the batchsize to 8, and set the learning rate to 0.0001.

heading: 4.2 MAIN RESULTS
text: In this section, we compare the algorithm proposed with other one-stage and two-stage algorithms on the VOC data set. The comparison results are shown in Table 4 and all the data has been published(Wang et al., 2019).
It can be seen from Table 4 that the algorithm proposed in this paper has a higher mAP on the VOC data set than most algorithms in recent years. It is 1.3% higher than PFPNet-R512(Kim et al., 2018). But it is 1.2% lower than IPG RCNN(Liu et al., 2020). This is largely due to the poor backbone and smaller image size, making the detection performance slightly worse than IPG RCNN(Liu et al., 2020). If we test the algorithm with multi-scale method, the mAP on the VOC data set can reach 85.1%, which is higher than all comparison algorithms.
At the same time, we reproduce the results of several state-of-the-art detectors on the VOC data set and compare them with the algorithm proposed in this paper to verify the effectiveness of our algorithm. The results are shown in the following Table 5.
It can be seen from Table 5 that the algorithm proposed in this paper has absolute advantages in AP and AR on tiny objects. The algorithm in this paper is 3.9% higher than YOLOV4(Bochkovskiy et al., 2020) (16.9%vs.13%), which has the highest APs among comparison algorithms. Compared with the RefineDet(Zhang et al., 2018), our proposed algorithm are 9.2% (29.4% vs. 20.2%) higher on ARs but 1.5% lower on APm. Meanwhile, the algorithm proposed in this paper has the highest AR of middle targets, which shows strong detectability for middle targets. We can see that the algorithm proposed in this paper has great advantages in detecting tiny objects. Both the AP and the AR of tiny objects perform well, which are better than most detection algorithms.

heading: 5 SUMMARY
text: We propose a composite structure of FPN, which contains a context augmentation module and a feature refinement module. The context augmentation module leverages the dilated convolution to extract the context information of different receptive fields and then integrates it into FPN to improve the context information of the tiny objects. The feature refinement module combines spatial adaptive fusion and channel adaptive fusion to suppress conflicting features from the dimensions of channel and space to highlight useful features. In addition, a copy-reduce-paste data enhancement method of tiny objects is proposed to prevent imbalance in training. Through experimental results, we can see that the tiny object detection network proposed in this paper performs well on the VOC data set. More results and Code are available at https://github.com/xiaojs18/ Object-Detection/tree/main/smallObjDetection.

Review summaries:
The paper aims at the tiny object detection and point out the issues are small context feature, semantic feature conflicts, and less tiny objects in training data.  To solve the aforementioned problems, authors introduce context augmentation module (CAM), design a feature refinement module, and adopt data-augmentation manner in training process. Experiments are conducted on PASCAL VOC dataset to validate the proposed methods and modules.