# Learning Motion Constraint-Based Spatio-Temporal Networks for Infrared Dim Target Detections
## 2 Materials and Methods
### Training Phase
#### 2.1.4 Extracting Deep Spatial Features Module  
After being processed by the spatio-temporal constraint module, the input frames are weighed by the temporal relationship, as shown in Figure 8. The left shows the processing of the single-frame image. The local contrasts of the target and the background are very similar. It is indistinguishable in the spatial dimension alone. The right indicates that the model with spatio-temporal constraints has superior results. For the single-frame method based on the local gradient, segmenting the target effectively is difficult when the target is very small and lacks texture features. The feature pyramid network (FPN) module has been applied extensively to universal target detection in recent studies. It extracts multi-scale features for fusion, thereby improving detection accuracy. Motivated by that, the DFM is joined behind the spatio-temporal module. Statistically, the size of the target may change between 2 \(\times\) 2 and 7 \(\times\) 7, and the general convolutional kernel and pooling operation might lose information. Hence, we chose feature maps with areas of 8, 16, and 32 scales. The fusion process is performed by up-sampling and down-sampling between different scales. Notice that we used 3D-CNN instead of the commonly used 2D-CNN. Since the 3D structure contains a temporal dimension compared to the 2D convolution, it is more appropriate for multiple-frame detection algorithms. Figure 9 shows a group of feature maps at different scales. Figure 8: A comparison of the single-frame-based and multiple-frame-based detection. It can be seen that multiple-frame-based detection offers more advantages. Figure 9: Multi-scale feature heat map: (**a**) 8 \(\times\) 8 feature maps; (**b**) 16 \(\times\) 16 feature maps; (**c**) 32 \(\times\) 32 features. It can be seen that large scale feature maps are sensitive to small targets, while small scale feature maps are effective for large-sized objects such as backgrounds. The loss function in this paper comprised two parts: coordinate regression loss and confidence loss. Equation (1) is the coordinate regression loss referenced in CIoU [29] (Complete IoU Loss; IoU: Intersection over Union). It evaluates the area intersection ratio, the distance relationship, and the aspect ratio's similarity between predicted and true coordinates. Equation (4) is the confidence loss [30], which is drawn upon to match with the former. It estimates the beingness of an object and picks out the best matching coordinate. Equation (6) indicates that the two component losses are linearly integrated during training. \[L_{CIoU(box_{pre},box_{gt})}=1-IoU\big{(}box_{pre},box_{gt}\big{)}+\frac{ \rho^{2}\big{(}box_{pre},box_{gt}\big{)}}{c^{2}}+\alpha v \tag{1}\]  
\[v=\frac{4}{\pi^{2}}(\arctan\frac{w^{gt}}{h^{gt}}-\arctan\frac{w}{h})^{2} \tag{2}\]  
\[\alpha=\frac{v}{(1-IoU)+v} \tag{3}\]  
\[L_{con}=\overset{\wedge}{y}\text{log}(y)+(1-\overset{\wedge}{y})\text{log}(1-y) \tag{4}\]  
\[y=P*IoU\big{(}box_{pre},box_{gt}\big{)} \tag{5}\]  
\[loss=\sum_{i}\sum_{j}L_{iCIoU(box_{pre},box_{gt})}^{j}+\sum_{i}\sum_{j}L_{iCon }^{j} \tag{6}\]  
In Equation (1), boxpre and boxgt, respectively, mean the network prediction and true coordinates. Part 1 - IoU (boxpre, boxgt) evaluates the overlapping area of the network prediction and true coordinates. \(\rho\) (boxpre, boxgt) computes the Euclidean distance. C represents the diagonal length of the minimum bounding box, which encircles boxpre and boxgt. The second part computes the distance of center point between the prediction and the true label. \(\upsilon\) compares the consistency of the aspect ratio. \(\upsilon\) is the width of the prediction box, and h is the height. Accordingly, wgt and hgt are the parameters of the true box. \(\alpha\) is a positive coefficient computed by \(\upsilon\) and IoU. The third part is used to control the convergence of the width and height as quickly as possible. Equation (4) is the confidence. It is evaluated by Equation (5), where \(p\) denotes the probability of the object.