# Learning Motion Constraint-Based Spatio-Temporal Networks for Infrared Dim Target Detections
## 1 Introduction
Widely used multi-frame algorithms take advantage of serial correlations. However, they consume too many computer resources. The calculation of fixed parameters and the sensitivity to bad points limit the effectiveness of traditional algorithms. _B. Deep learning methods_  
Early methods that relied on gradient operation were gradually surpassed by popular convolutional neural networks. Recent approaches are inclined toward the deep learning model. The R-CNN (Region-CNN, CNN: convolutional neural network) family [12] is used on behalf of the two-stage detector. The candidate region is generated first for classification and regression. The Yolo (You Only Look Once) family [13; 14], the SSD (Single Shot MultiBox Detector) [15], and the RetinaNet [16] are representatives of the single-stage detector. In contrast, coordinates and categories regressed directly without candidates. The single-stage methods greatly improve detection speeds. However, their convolution layers are too deep for the dim target. The high-level features are not prominent in long-range infrared frames. In light of the above problems, researchers have contributed large efforts. [PERSON] et al. [17] improved the SSD to fuse infrared gray-temperature data. Still, it is based on a single frame and cannot solve occasional occlusion. [PERSON] et al. [18] combines the full convolution regression network (FCRN) with graph matching to achieve robust detection. Meanwhile, they use the optical flow to obtain motion information. [PERSON] et al. [19] designed an end-to-end model inspired by the denoising autoencoder, which treats the small target as the \"noise\" and transforms detections into a denoising task. However, it still causes terrible false alarms. [PERSON] et al. [20] proposed a single-shot detector. It comprises an image filtering module (IFM) and infrared small target detection module (ISTDM). The IFM aims to enhance valid information and suppress noise. The ISTDM is an efficient convolutional neural network for improving small target detection tasks. [PERSON] et al. (2018) focused on applying a deep neural network to classify the proposal regions rather than pre-processing to reduce the false alarm. [PERSON] et al. (2019) attempted to take small infrared target detection as a classifying task. They exploited synthetic data to train the proposed deep-learning network. Their experiments show that the choice of SNR affects the performance of deep networks. [PERSON] et al. (2019) combined Fully Convolutional One-Stage Object Detection (FCOS) with traditional filtering to enhance the target and suppress the background. The method takes multiple frames as the input of the network but the structure remains single-frame-based. [PERSON] et al. (2019) employed the Mask R-CNN model and transfer learning to extract the feature of the small target. However, the method ignores the temporal relation of the moving target. The discussion shows the usefulness of deep learning models. However, they still have interruption risks during detection when applied in practice, such as uncertain occlusion and redundant calculations. To overcome the existing defect and be motivated by the data-driven proposal, a learning motion constraint network is proposed in this paper. The processing flow of the proposed method is illustrated in Figure 1. In the training phase, spatio-temporal features are learned by stacked Conv-LSTM cells (spatio-temporal constraint module, called STM). Then, the multiscale feature fusion is established by 3D-Conv cells (deep spatial features module, called DFM), which could broaden receptive fields and extract more robust features between the target and the background. The last full connection cell generates the network prediction. In the inference phase, a variable interval search strategy called a state-aware module (state-aware module, called STAM) is proposed. It decides that the network searches for global images only if the target is lost due to fast motion, uncertain obstruction, or frame loss. This submodule highly increases the speed of detection.