# Dual Enhancement Network for Infrared Small Target Detection
## 3 The Proposed Dual Enhancement Network
### Residual Sparse Enhancement Module  
Inspired by [27], we considered propagating global context information in a sparse manner, which only propagates selective key pixels between adjacent levels. To achieve this, we designed a sparse semantic propagation (SSP) module, and we appended it to the FPN [28] framework to capture multi-level feature advantages. In the SSP module, we first selected the key pixels and then spread sparse semantic information between adjacent levels. The following is the detailed process. Figure 2: Framework of our DENet, which consists of two distinct designs: (1) residual sparse enhancement (RSE) module and (2) spatial attention enhancement (SAE) module. Above: Pipeline of sparse semantic propagation (SSP) module. As is shown in the upper part of Figure 2, the most salient part of the input features and key pixel indexes can be extracted through the following two steps: (1) generate the salient map; (2) generate sampled pixel indexes. Then, sparse semantics are propagated between the adjacent levels shown in Equation (3). For the first step, we convert the channel dimensions of two input feature maps to be consistent, downsample the high-resolution feature \(F_{l-1}\) to match the low-resolution feature \(F_{l}\), and obtain a new resized feature denoted as \(\widetilde{F}_{l-1}\). Then, we concat the \(\widetilde{F}_{l-1}\) and the converted \(F_{l}\), and we exploit one \(3\times 3\) convolution following with the sigmoid function to generate the saliency map \(M_{l}\). The saliency map \(M_{l}\) can be defined as  
\[M_{l}=\text{Sigmoid}\Big{(}\text{conv}_{l}\Big{(}\text{Concat}\Big{(}F_{l}, \widetilde{F}_{l-1}\Big{)}\Big{)}\Big{)}, \tag{1}\]  
For the second step, the pixel index generator takes \(F_{l}\) and \(M_{l}\) as inputs. To gain the most salient pixels, we employ adaptive max pooling on the saliency map \(M_{l}\). Then, as is shown in Equation (2), we multiply the saliency map on \(F_{l}\) with the residual structure to enhance the saliency of the foreground objects:  
\[F_{l}^{s}=\text{Maxpool}(M_{l})\times F_{l}+F_{l}, \tag{2}\]  
We choose the salient indexes from \(Maxpool(M_{l})\) and denote them as \(I_{s}\) for short. To sum up, the pixel index generator takes \(F_{l}\) and \(M_{l}\) as inputs and generates the salient indexes \(I_{s}\) and the salient map \(F_{l}^{s}\) for a sampler. The complete operation of the pixel index generator is illustrated in Figure 2. With the salient indexes \(I_{s}\) gained from the pixel index generator, we sample pixels from the input feature map \(F_{l-1}\in R^{\mathbb{C}\times H\times W}\) and salient map \(F_{l}^{s}\in R^{\mathbb{C}\times H/2\times W/2}\). For each selected pixel \(\hat{p}\), we extract the pixel-wise feature \(f\) on both adjacent input features, and we then propagate those sampled pixels from the top to the bottom. The specific propagation process is shown in Equation (3):  
\[f_{l-1}(\hat{p})^{r}=A(f_{l-1}(\hat{p}),f_{l}(\hat{p}))f_{l}(\hat{p})+f_{l-1}( \hat{p}), \tag{3}\]  
We begin by flattening \(F_{l}^{s}\) into a two-dimensional vector. Subsequently, we utilise the salient index \(I_{s}\), generated by the pixel index generator, to gather high-resolution salient features from \(F_{l}^{s}\). The resulting features are then reshaped to match the dimensions of \(I_{s}\). Similarly, we flatten \(F_{l-1}\) into a 2D vector. The corresponding salient index \(I_{low}^{s}\) of \(F_{l-1}\) is determined based on the dimensional relationship between \(I_{s}\) and the upper and lower features. Utilizing this index, we gather the low-resolution salient features from \(F_{l-1}\), adjusting their shapes to align with \(I_{low}^{s}\). Finally, we transform both the high-resolution and low-resolution salient features into \(f_{l}(\hat{p})\) and \(f_{l-1}(\hat{p})\) through size transformation, where A represents the affinity function, \(f_{l}\) denotes the sampled feature at level \(l\) for the salient region, and \(f_{l-1}^{r}\) denotes the refined affinity feature.