# Attention Mechanism Used in Monocular Depth Estimation: An Overview
## 2 Preliminary
### Attention  
The concept of attention is inspired by the biological system. For instance, in the image classification task, the salient components are more relevant to the object category, and should be paid more attention after a glimpse of the holistic scene. In the machine translation task, some words may have more influence to the output. Therefore, each component of the source may have different impacts on the target, and should not be equally treated. The attention mechanism is able to not only model the relevance between the source and the target, but also generate new representations according to the weights of each component of the source. Technically, the attention model calculates coefficients using a query (\(Q\)) and a set of keys (\(K\)) and generates new representations by combining values (\(V\)) weighted by the coefficients. A generalized attention model is defined as Equation (3). For details, please refer to [25]. \[Attn(Q,K,V)=\sum_{i}p(\sigma(k_{i},Q))\times v_{i} \tag{3}\]  
where \(p\) is distribution function and \(\sigma\) is alignment function. [PERSON] et al. firstly proposed an attention model to address the issue of the machine translation task [12]. Recently, the attention mechanism cooperating with CNN was popular in the computer vision tasks to capture the long-range context [26]. In the MDE task, attention can be applied over the space axis or the channel axis. Additionally, the source and the target may be the same sequences or the different sequences. Thus, we classify the attention mechanism in the MDE task as four categories: channel self-attention (CSA), channel cross-attention (CCA), spatial self-attention (SSA), and spatial cross-attention (SCA). Channel self-attention. The attention map is generated over different channels of feature maps extracted by an encoder. Channel self-attention fuses the information of several channels since each channel indicates different aspects of the input image. [PERSON] et al. built a squeeze-and-excitation (SE) block [27], as shown in Figure 1, to recalibrate the relationship of channels. A global average pooling is firstly exerted over the whole image to produce the global dependency in the squeeze stage; a sigmoid gating function is then used to generate an attention map in the excitation stage. [PERSON] et al. proposed the convolutional block attention module (CBAM), consisting of a channel attention module and a spatial attention module to enhance the performance of CNN [28]. The channel attention module of CBAM is shown in Figure 2. The input feature map \(F\in R^{C\times H\times W}\) is pooled parallel within the \(H\times W\) space by average-pool and max-pool operations to obtain two 1D vectors of \(C\) variables. Then, these two vectors are fed into full connection layers to generate the channel attention map, i.e., the channel weights. At last, the new representation is output by multiplication of channel weights and the original inputs. Compared with SE, we see that CBAM uses not only average pooling, but also maximum pooling. In addition, the channel attention module is followed by a spatial attention module in CBAM. Channel cross-attention. Unlike channel self-attention, the attention map of channel cross-attention is generated by the interactions of groups of feature maps extracted by different encoders. [PERSON] et al. utilize the channel cross-attention module, which is shown in Figure 3a, to capture global dependencies among channels by leveraging cross-attention across channel tokens of multi-scale encoder features [29]. To begin, each token is subjected to a layer normalization operation. The tokens are then concatenated along the channel dimension to form keys and values. Finally, the projected queries, keys, and values are fed into cross-attention. Spatial self-attention. The input of the attention module is an input image or a specific feature map. We call it spatial self-attention when the \(Q\), \(K\), and \(V\) are of the same inputs. [PERSON] et al. proposed an attention implementation, which is shown in Figure 4[28].