# SDebrisNet: A Spatial-Temporal Saliency Network for Space Debris Detection
## 2 Space Debris Detection
### Temporal Feature Extraction  
The mainstream temporal feature extraction approaches mainly include 3D- convolution [48], recurrent neural network [39; 49], optical flow [35; 50] and attention-based mechanisms [34; 37; 51]. However, the computational cost of the 3D convolution operation is usually expensive, which is not fit for the space-based platform with limited computational resources. For the recurrent network, the convolutional memory units, such as convolutional long short-term memory (ConvLSTM) and convolutional gated recurrent unit (ConvGRU) lead to high computational costs and high memory costs. The optical flow method requires an additional network branch to obtain optical flow information, which means it is not a real end-to-end network. Moreover, when the camera moves due to the incident jitter of the satellite video, the movement of space debris may be too small  
Figure 4: Spatial feature enhancement module. compared to the background, resulting in a weak correlation between salient objects and movement information contained in optical flow. Recently, non-local self-attention-based methods [34; 39; 52] have been widely applied to the video salient object detection to model the temporal features. They can capture long-range dependencies in videos by establishing pair-wise relationships among feature elements in consecutive frames. However, they cannot be applied in space debris detection in videos (directly) for the following reasons. Firstly, they focus on capturing motion-independent global contexts instead of motion cues, which are not suitable for detecting moving space debris because the moving objects tend to be salient and are more attractive to human attention [53]. Secondly, the computational and memory costs are expensive because the FLOPs (floating point operations) and memory consumption are quadratic functions with respect to the number of frames and the feature resolution, which are not fit for the limited computations and storage resources of space-based applications. Motivated by the constrained self-attention (CSA) operation in [51], which can model motion cues more efficiently, a parallel structure composed of four CSA operations was designed as the TFM to extract the temporal features. Given the high-level spatial features \(C_{S}\in\mathbb{R}^{\mathbb{C}\times N\times H\times W}\) generated by SFM, they are split into four groups, i.e., \(C_{S^{\prime}}\in\mathbb{R}^{\widehat{\mathbb{C}}\times N\times H\times W}\). Each \(C_{S^{\prime}}\) is fed into a CSA network with different dilations and window sizes. The four resulting features \(C_{T^{\prime}}\) are concatenated as temporal features \(C_{T}\). The structure of the CSA network can be found in Figure 5. The \(C_{S^{\prime}}\) is projected into three subspaces, i.e., query space \(\mathbf{Q}\), key space \(\mathbf{K}\), and value space \(\mathbf{V}\), using \(1\times 1\times 1\) convolution as the linear function. Based on the objects sharing similar positions in adjacent frames, when a feature element \(q\in\mathbb{R}^{\widehat{\mathbb{C}}\times 1\times 1\times 1}\) is queried in the position \((n,h,w)\) of \(\mathbf{Q}\), where \(1\leq n\leq N,1\leq h\leq H,1\leq w\leq W\), the constrained neighborhood \(S_{q}\) surrounding \(q\) in \(\mathbf{K}\) is selected to compute affinity by the dot-product operation. \(S_{q}\) can be formulated as  
\[S_{q}=\{K(n^{\prime},h^{\prime},w^{\prime})\}_{n^{\prime}=1,h^{\prime}=h-dt,w^ {\prime}=w-dr}^{N,h+dr,w+dr} \tag{2}\]  
where \(S_{q}\in\mathbb{R}^{N(2r+1)^{2}\times\frac{\mathbb{C}}{4}}\). \(K(n^{\prime},h^{\prime},w^{\prime})\in\mathbb{R}^{\widehat{\mathbb{C}}\times 1 \times 1}\), \(1\leq n^{\prime}\leq N,1\leq h^{\prime}\leq H,1\leq w^{\prime}\leq W\). \(r\) and \(d\) are the size and dilation of the sliding window, respectively. Thus, the size of the constrained neighborhood can be determined by different sliding window sizes \(r\), dilation \(d\), and the number of video frames \(N\).