[11] designed four network fusion structures based on Faster R-CNN to combine infrared and visible information and find out the optimal fusion strategy. [PERSON] et al. used two region of interest (ROI), pooling at the middle and bottom layers of the feature exactor, to obtain information at different scales. The results showed that this technique could effectively reduce the miss rate of pedestrian detection [12]. However, in Faster R-CNN, the region proposal network (RPN) only uses the last layer of the feature extraction network to generate proposals, which is not suitable for multi-scale object detection. Although more anchors with different scales can be set up to compensate, the cost of computation is highly increased. Moreover, Faster R-CNN only uses the coarsest feature map to classify, which is not capable of capturing the representation of small targets. In order to improve the performance of Faster R-CNN, there have been many efforts in the generic object detection pipeline. Some researchers integrate the low-level features with the high-level ones to makes the network more sensitive to small objects [13; 14]. SSD [15], RSSD [16], and DSSD [17] generate proposals and predict objects on each layer of backbone to improve the prediction accuracy of multi-scale objects. FPN [18], RON [19], and Mask R-CNN [20] combine coarse to fine features to capture richer multi-scale representation. However, these methods inevitably enlarge the feature dimension, which leads to a rise of computational cost. In order to balance efficiency and performance, some researchers [21; 22; 23] have employed multi-scale ROI pooling to combine the output of different hierarchies. The network can then obtain multi-scale features simultaneously. Other studies have focused on changing the receptive field of feature maps to obtain multi-scale information without significant increase in computational complexity. DetNet [24] and RFBNet [25] add dilated convolution into the feature extraction network to increase the density of the feature map. The experiment results show that the performance of these methods is competitive with the mainstream ones. Another issue that affects the accuracy of object detection on infrared images is the lack of high-quality datasets. The common solution is data augmentation, for example, flipping, changing the contrast, blurring, cropping, and mirroring. However, in the field of object detection, there is little research on it. In Reference [15], the Pascal VOC dataset was randomly cropped, mirrored, and scaled to further improve the mean average precision of SSD for small objects. In Reference [14], an algorithm randomly selected angles, then the images and annotations were rotated accordingly. In Reference [26], an algorithm randomly selected a rectangular region in the image, and replaced its pixels with random values. Multi-scale occlusion was then generated. To summarize, these existing data augmentation methods for object detection have the following shortcomings. Firstly, these algorithms randomly transform every image without considering the proportions of different objects in the dataset. It is not helpful to solve non-uniform class distribution. Secondly, since these methods take the whole image to rotate, clip, scale, or erase, they are not able to solve non-uniform class distribution when different kinds of objects exist in the image. Thirdly, in the process of augmentation, these algorithmsdo not take into account the spatial relationship between the objects in the image. It is easy to lead to the loss of image information. In the field of image classification, [PERSON] et al. put forward the method of under-sampling, which can effectively expand the amount of minor category data without losing the original data [27]. However, these methods have difficulty balancing the proportions of different categories while expanding the object detection dataset, especially when the proportions of categories vary greatly.