CONTEXT AUGMENTATION AND FEATURE REFINEMENT NETWORK FOR TINY OBJECT DETECTION

Tiny objects are hard to detect due to their low resolution and small size . The poor detection performance of tiny objects is mainly caused by the limitation of network and the imbalance of training dataset . A new feature pyramid network is proposed to combine context augmentation and feature refinement . The features from multi-scale dilated convolution are fused and injected into feature pyramid network from top to bottom to supplement context information . The channel and spatial feature refinement mechanism is introduced to suppress the conflicting formation in multi-scale feature fusion and prevent tiny objects from being submerged in the conflict information . In addition , a data enhancement method called copy-reduce-paste is proposed , which can increase the contribution of tiny objects to loss during training , ensuring a more balanced training . Experimental results show that the mean average precision of target targets on the VOC dataset of the proposed network reaches 16.9 % ( IOU=0.5:0.95 ) , which is 3.9 % higher than YOLOV4 , 7.7 % higher than CenterNet , and 5.3 % higher than RefineDet . 1 INTRODUCTION . As an challenge in the field of target detection , tiny object detection is widely used in vision tasks such as autonomous driving , medical field , drone navigation , satellite positioning , and industrial detection . In recent years , object detectors based on deep learning have made great progress ( Tong et al. , 2020 ; Feng et al. , 2020 ) . One-stage algorithms represented by ( Redmon & Farhadi , 2018 ; Zhang et al. , 2020 ; Liu et al. , 2016 ) can directly get access to the target of interest through the forward convolutional neural network with a fast speed . However , two-stage algorithms ( He et al. , 2017 ; Ren et al. , 2016 ) obtain the RoI ( Region of Interest ) based on the generated candidate boxes , which has higher accuracy . Although these target detection algorithms have made great progress in precision and speed , their performance is still very unsatisfactory when detecting tiny targets ( less than 32×32 pixels ) . On most public data sets , the detection precision of tiny objects is even less than half of that of larger targets ( Liu et al. , 2016 ) . Therefore , tiny object detection still has a lot of room for improvement . The poor performance of tiny object detection is mainly caused by the limitations of the network itself and the imbalance of training data ( Kisantal et al. , 2019 ) . To obtain solid semantic information , modern detectors try to stack more and more pooling and down-sampling operations so that tiny object features with few pixels are gradually lost in forwarding propagation ( Liu et al. , 2021 ) , limiting the detection performance of tiny objects . FPN ( Lin et al. , 2017 ) can alleviate the problem of information diffusion to a certain extent ( Redmon & Farhadi , 2018 ; Liu et al. , 2016 ) by fusing low-resolution feature maps with high-resolution feature maps horizontally . However , fusing the information of different densities directly will cause semantic conflicts , which limiting the expression of multi-scale features and making tiny objects submerged in conflicting information easily . At the same time , in the current classic public data set , the number of annotations of tiny objects is much less than that of larger targets ( Chen et al. , 2020 ) . Therefore , the convergence direction of the network is continuously leaning toward larger targets during training , resulting in poor performance of tiny objects . Consequently , we believe that it is feasible to improve the detection rate of tiny objects from the above two aspects . To solve the problem of feature dispersion of tiny objects and semantic differences between layers , this paper proposes a new feature pyramid composite neural network structure that combines context augmentation and feature refinement . The proposed algorithm framework is shown in Figure 1 . It is combined with context augmentation module ( CAM ) and feature refinement module ( FRM ) . CAM merges multi-scale dilated convolution features to obtain rich context information for feature augmentation . FRM introduces a feature refinement mechanism in the dimensions of the channel and space to suppress conflict information and prevent tiny objects from being submerged in conflict information . Simultaneously , to ensure that the network will not lean towards larger targets during training , a method called copy-reduce-paste is proposed to increase the contribution to the loss of tiny objects in training . We train and test on the standard public data set PASCAL VOC , and verify the effectiveness of out method for detecting tiny objects through comparative experiments and ablation experiments . The algorithm proposed in this paper achieves a precision of 83.6 % ( IOU=0.5 ) on the VOC data set , which is higher than most comparison algorithms , and the precision of tiny objects reaches 16.9 % ( IOU=0.5:0.95 ) , which is higher than YOLOV4 , CenterNet and other cutting-edge networks . 2 RELATED WORK . 2.1 OBJECT DETECTOR BASED ON DEEP LEARNING . As a fundamental computer vision task , target detection contains both classification and localization , which can be regarded as a regression problem . In the early days , hand-designed features were widely applied to target detection . However , hand-designed feature is a kind of shallow feature , and it is gradually replaced after the appearance of the CNN-based features . R-CNN ( He et al. , 2017 ) , as the pioneering work of two-stage algorithms , employs prior boxes of different size to match targets of different size and then selects candidate regions through CNN . To reduce the training time , FastRCNN ( Ren et al. , 2016 ; Xiao et al. , 2020 ) extracts the feature map of the entire image , and then spatial pyramid pooling and RoI ( Region of Interest ) pooling are used to generate regional features and to filter candidate regions . To further improve the precision of tiny object , EFPN ( Deng et al. , 2021 ) proposes a super-resolution feature pyramid structure to amplify tiny object features . Compared with two-stage networks , one-stage networks have a faster speed but lower precision . SSD ( Liu et al. , 2016 ) puts anchor boxes densely on the image to obtain the target boxes , and meanwhile , it makes full use of features of different scales to detect smaller targets . YOLOV3 ( Redmon & Farhadi , 2018 ) chooses to detect large , medium , and tiny objects separately based on three outputs of the feature pyramid , which significantly improves the detection performance of tiny objects . ( Zhang et al. , 2020 ) introduces a high-resolution attention mechanism to FPN to mine the most useful information of tiny targets . This paper chooses YOLOV3 ( Redmon & Farhadi , 2018 ) as the baseline and makes improvements on this basis . RefineDet ( Zhang et al. , 2018 ) introduces a new loss function to solve the imbalance between simple samples and difficult samples . Recently , detectors based on anchorfree architecture are becoming more and more popular ( Zhao et al. , 2019 ) . Although target detection algorithms are constantly developing and replacing , there is no big breakthrough in the field of tiny object detection , and the detection precision of tiny objects stays low . 2.2 MULTI-SCALE FEATURE FUSION . Using multi-scale features is an effective method to improve the detection precision of tiny objects . SSD ( Liu et al. , 2016 ) is the first attempt to predict the location and category of targets with multiscale features . FPN ( Lin et al. , 2017 ) merges adjacent feature maps with different grains from top to bottom , which can improve the expressive ability of features greatly . A large number of variant structures similar to FPN ( Lin et al. , 2017 ) have emerged . PANet ( Liu et al. , 2018 ) adds extra bottomup connections based on FPN ( Lin et al. , 2017 ) to transfer information from the lower layer to the upper layer more efficiently . NAS-FPN ( Ghiasi et al. , 2019 ) found a new connection method through neural architecture search technology . BiFPN ( Tan et al. , 2020 ) improved the connection method of PANet ( Liu et al. , 2018 ) to make it more efficient and introduced a simple attention mechanism at the connection point . Although the structures mentioned above have greatly improved the multi-scale expression ability of the network , they have ignored the existence of conflict information between features of different scales , and the lack of context information may hinder the further improvement of performance , especially for tiny objects , which is easy to be submerged in conflict information . This article fully considers the impact of conflict information and context information on detection precision . 2.3 DATA AUGMENTATION . Preprocessing of the training set has always been an indispensable part of deep learning , such as rotation , deformation , random erasure , random occlusion , illumination distortion , and MixUp . In recent days , several data enhancement methods for tiny objects have been proposed . ( Chen et al. , 2020 ) regards loss as a kind of feedback . And four images were scaled to the same size and stitched together to enhance the performance of tiny object detection under the guidance of feedback . Unlike ( Chen et al. , 2020 ) , ( Yu & Koltun , 2015 ) scales 4 images to different sizes and stitched them into one . ( Kisantal et al. , 2019 ) tried to achieve tiny object data augmentation by copying and pasting tiny objects back to original images . This method can only increase the number of tiny objects but not the number of training images containing tiny objects . It will also cause the imbalance of training to a certain extent . Because larger targets are widely distributed in each batch of training , this paper guarantees the tiny objects ’ contribution to the loss in eache batch of training , making the training more balanced . 3 PROPOSED METHODS . This chapter will introduce our tiny object detection network in detail . As we can see in Figure 1 , { C2 , C3 , C4 , C5 } represent different levels after input image being down-sampled by { 4 , 8 , 16 , 32 } times . { F1 , F2 , F3 } are denoted as newly generated feature levels corresponding to { C3 , C4 , C5 } by a layer of convolution , and C2 is discarded because of a mess of noises . { L1 , L2 , L3 } are denoted as feature levels generated by FPN and { P1 , P2 , P3 } are denoted as the feature levels generated by FRM . The network is mainly composed of CAM and FRM . CAM is inspired by the mode that humans recognize objects . For example , it is difficult for human to distinguish a bird in a very high sky , but it is easy for human to distinguish when considering the sky as the context information . Therefore , we believe that context information is helpful for tiny object detection . CAM applies dilated convolution with different dilated convolution rates to obtain context information of different receptive fields , and injects it into FPN ( Lin et al. , 2017 ) from top to bottom to enrich context information . But it will introduce redundant information and conflicting information while sharing the information , because of the semantic differences among different levels of FPN ( Lin et al. , 2017 ) . Therefore , FRM is proposed to filter conflict information and reduce semantic differences . By fusing the features between different layers adaptively , the conflict information among layers is eliminated to prevent the tiny object features from being submerged in the conflict information . Simultaneously , in view of the small number of positive samples generated by tiny objects and the limited contribution to loss of tiny objects , a data augmentation method called copy-reduce-paste is proposed . Specifically , copy the larger targets in the training set , reduce them , and then paste them back to the original imge . During the pasting process , it is necessary to ensure that the pasted targets do not overlap with the existing targets . The above methods will be explained in detail in the following sections .

Summary: The paper aims at the tiny object detection and point out the issues are small context feature, semantic feature conflicts, and less tiny objects in training data.  To solve the aforementioned problems, authors introduce context augmentation module (CAM), design a feature refinement module, and adopt data-augmentation manner in training process. Experiments are conducted on PASCAL VOC dataset to validate the proposed methods and modules.