# Adaptive Frame Sampling and Feature Alignment for Multi-Frame Infrared Small Target Detection
## 3 Our Method
### Network Architecture
#### 3.2.1 Feature Extraction and Alignment Module  
After adaptive frame sampling is performed on a set of continuous infrared images, \(F=[F_{0},F_{1},\ldots,F_{T}]\) that contains \(T+1\) frames, a set of infrared images, \(\overline{F}=[\overline{F_{1}},\overline{F_{2}},\ldots,\overline{F_{N}}]\) that contains \(N\) frames is obtained as the input of the detection network, where \(\overline{F_{N}}\) is the frame to be detected. For each frame of the input network, its features are extracted using a shared weight backbone network. This backbone network was modified from MobileNetV3 [36]. In order to better extract the detailed information of infrared small target images, the number of downsampling operations was reduced to obtain a high-resolution feature map. The detailed structure is shown in Table 1. Except for the first row, each row of operation operators is connected in series to form the backbone network. Input in the first row represents the resolution and number of channels of the input feature map of the operation operator in that row, and Operator represents the operation operator, among which Conv2d,3 represents a two-dimensional convolution with a convolution kernel of 3 concatenated with a standard convolution block of batch normalization and an activation function. Bneck represents the basic bottleneck structure, using depthwise separable convolution [37]. CCA stands for cross attention module [38], which serves as the last layer of the backbone network, to strengthen the local important information of the spatial features obtained. Exp Size represents the number of channels of the feature map in Bneck after the first Dwconv. Out Channel represents the number of channels of the feature map after this operator. SE represents the channel attention network [39], True  
Figure 2: The process of adaptive frame sampling based on mutual information. indicates that the Bneck uses channel attention, and False indicates that channel attention is not used. NL represents what activation function the operator uses, where HS represents the HardSwish activation function and RE represents the ReLU activation function. Stride represents the step size of the operator. If the step size is 1, the resolution of the input and output feature maps will remain unchanged. If the step size is 2, the feature map will be downsampled. The feature alignment module is shown in Figure 4. Inspired by the literature [40], the feature map to be aligned \(f_{i}\) and the current feature map \(f_{N}\) are concatenated and fed into the homography estimation network to obtain the corresponding homography flow matrix. Then, a warp operation is performed on the \(f_{i}\) to obtain an aligned feature map. The aligned feature map \(f_{i}^{\prime}\) is subtracted from the feature map \(f_{i}\) to obtain a differential feature map \(f_{s}\), and is concatenated with \(f_{i}^{\prime}\) as the output \(f_{out}\). In order to ensure the quality of alignment, we also introduce a triplet loss for the feature alignment module, which is explained in detail in Section 3.3. As shown in Figure 4, the estimator in the feature alignment module consists of the last three layers of ResNet18 [41], and finally eight weights are obtained through two fully connected layers. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Input Size** & **Operator** & **Exp Size** & **Output Channels** & **SE** & **NL** & **Stride** \\ \hline \(256^{2}\times 3\) & Conv2d,3 & - & 16 & - & HS & 1 \\ \(256^{2}\times 16\) & Bneck,3 & 16 & 16 & True & RE & 2 \\ \(128^{2}\times 16\) & Bneck,3 & 72 & 24 & False & RE & 1 \\ \(128^{2}\times 24\) & Bneck,3 & 88 & 24 & False & RE & 1 \\ \(128^{2}\times 24\) & Bneck,5 & 96 & 40 & True & HS & 2 \\ \(64^{2}\times 40\) & Bneck,5 & 240 & 40 & True & HS & 1 \\ \(64^{2}\times 40\) & CCA & - & 40 & - & - & - \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Network structure of backbone network. Figure 3: The architecture of the detection network.