# Saliency Guided DNL-Yolo for Optical Remote Sensing Images for Off-Shore Ship Detection
## 2 Methodology
### Saliency Tuned YOLONet  
Most of the existing methods only use convolution operations to extract features. However, non-local blocks are a popular self-attention module used to enhance the modeling ability of CNN and it shows good performance in visual applications. DNL module is an improved version of a non-local module. By using an independent softmax function and disentangling the embedding matrix, the learning complexity of the CNN is reduced, and a clearer salient area can be extracted. As a consequence, the DNL module is added on the architecture of the YOLOv4 network to combine both convolution and attention operations, which can further explore the salient features in image. Especially for ships that are prominent targets in the ocean background, it can exert a better detection performance. The main framework of the network model is illustrated in Figure 2.  
**DNL module. ** The DNL module is obtained by optimizing the non-local module, which is a popular module used to enhance the context modeling capabilities of traditional CNN. The calculation formula of the original non-local module is as follows:  
\[y_{i}=\sum_{j\in\Omega}\omega\big{(}x_{i},x_{j}\big{)}g(x_{j}) \tag{17}\]  
\[g\big{(}x_{j}\big{)}=W_{g}*x_{j} \tag{18}\]  
where \(\Omega\) denotes the all pixels of input feature map. \(g(\cdot)\) is the transformation function related to weight matrix \(W_{g}\), which is utilized to map a point to a vector. \(\omega(x_{i},x_{j})\) is the similarity function of pixel \(j\) (referred to as a center pixel) and other pixel \(i\), typically instantiated by an embedded Gaussian as:  
\[\omega\big{(}x_{i},x_{j}\big{)}=\sigma\Big{(}p_{i}^{\mathrm{T}}c_{j}\Big{)}= \frac{\exp\big{(}p_{i}^{\mathrm{T}}c_{j}\big{)}}{\sum_{l\in\Omega}\exp\big{(} p_{i}^{\mathrm{T}}c_{l}\big{)}} \tag{19}\]  
where, \(p_{i}=W_{p}x_{i}\) and \(c_{j}=W_{c}x_{j}\) denote the embedding of pixel \(i\) and \(j\), and \(\sigma(\cdot)\) is the softmax function. After that, the above formula can be transformed by a whitening dot product denoted as:\[\omega(x_{i},x_{i})=\sigma\left(\underbrace{\left(p_{i}-\mu_{p}\right)^{T}\!\left( c_{j}-\mu_{c}\right)}_{\text{\emph{pairwise}}}+\underbrace{\mu_{p}^{T}c_{j}}_{ \text{\emph{unary}}}\right) \tag{20}\]  
where the \(\mu_{p}\) and \(\mu_{c}\) are the averaged embedding over all of the pixels. The first _whitened_ dot product term represents the pure _pairwise_ relation between a pixel \(i\) and a pixel \(j\), and the second term represents the _unary_ relation where a pixel \(j\) has the same impact on all pixels \(i\). As a consequence, the DNL module divides the original formula into two terms, paired terms and unary terms, by disentangling matrix. One of them then learns the relationship within the area, and the other learns the salient boundary. The separation of these two items can then reduce the mutual interference, thereby achieving a better feature extraction effect. The specific frame is shown in the dashed box in Figure 2. For an input feature map of \(C\times H\times W\), two \(1\times 1\) convolutions \(w_{k}\), \(w_{q}\) is utilized to extract features firstly. Then they are calculated via a _whitening_ dot product. Finally, the output features of \(HW\times HW\) are obtained through the softmax function. In another branch, an independent \(1\times 1\) convolution \(w_{m}\) is used to extract features, and the softmax with expand operations are connected to obtain the same size feature as the above. At the end, we added the previous two results, and then a dot product with a \(1\times 1\) convolution \(w_{m}\) of the original input feature in order to obtain the final output. **DNL-added YOLONet. ** The YOLOV4 network is a recent developed optimization framework in the YOLO family for target detection. The YOLOV4 network model mainly includes five basic components: CBM, CBL, Res Unit, CSPX, and SPP, as shown in Figure 4. The CBM component is composed of convolution, batchnorm, and mis activation functions, and is the most basic structural unit.