# Attention Mechanism Used in Monocular Depth Estimation: An Overview
## 3 Attention Mechanism in MDE Methods
### CNN-Based Attention Methods
#### 3.2.2 Self-Supervised Training  
Recently, self-supervised approaches were predominant in the community of MDE. Self-supervised MDE methods exploit geometric constraints contained in the video sequence or stereo pairs as training loss, thus removing the expensive cost of sample labelling. To calculate photometric error from video sequences, pose networks are often used to estimate extrinsics of cameras to obtain wrapped images. [PERSON] et al. reported a self-supervised method in which channel and spatial attentions are embedded at the bottleneck of an encoder-decoder architecture [41]. Similar works include [42; 43], in which channel and spatial attention blocks are parallel or sequentially inserted at the bottleneck. The channel and spatial attention modules can also be inserted into any stage of the decoder [44]. Similarly, [PERSON] et al. also added a channel attention block into each stage of the decoder in a U-net structure to adjust the weights of channels inputted from the skip connections [45]. [PERSON] et al. proposed a discrete disparity prediction method in which a ResNet encoder followed by a spatial self-attention module is used to predict low-resolution disparity [46]. The low-resolution disparity is then processed by a multi-scale decoder to yield high-resolution disparity. A temporally consistent depth (TC-Depth) prediction method was proposed by [PERSON] et al. [31], in which a spatial-temporal attention block is placed at the bottleneck to aggregate geometric and sequential consistencies. Unlike those of SE and CBAM, TC depth adopts physical distance between points of 3D space to measure the spatial attention map defined as follows:  
\[Att_{i,j}=\exp\left(-\frac{\parallel P_{i}-P_{j}\parallel_{2}}{\sigma}\right) \tag{5}\]  
where \(P_{i}\) and \(P_{j}\) are points of 3D space back-projected by depth \(d_{i}\), \(d_{j}\), and coordinates of pixels \(C_{i}\) and \(C_{j}\) of a 2D image, and \(K\) indicates the intrinsics of the camera. \[P_{i}=K^{-1}(d_{i}\cdot C_{i}),\ P_{j}=K^{-1}\big{(}d_{j}\cdot C_{j}\big{)} \tag{6}\]  
Another contribution of the work is temporal attention, which uses consecutive frames as \(Q\) and \(K\) to calculate attention weights along the temporal axis. We categorize this mechanism as one of the kinds of of spatial cross-attention since the \(Q\) and \(K\) are from different representations. [PERSON] et al. proposed a channel-wise attention-based depth prediction method in which a structure perception module and a details emphasis module are utilized to aggregate and recalibrate the channel features [47]. [PERSON] et al. proposed a lightweight network for MDE using stereo pairs [48]. In their scheme, the discriminator of generative adversarial networks (GANs) [49] is used to distinguish the warped image and the original image. The generator of GANs is constructed by U-net networks with attention blocks embedded in the decoder. The proposed attention block consists of global average pooling operations and a group of dilated convolutional operations with an increasing receptive field to aggregate multi-scale features and recalibrate the channel-wise information. The attention model of [48] is shown in Figure 13.  
[PERSON] et al. designed a multi-level dual attention network (MLDA-Net) for self-supervised MDE [50]. The input image is firstly down-sampled to extract multi-level features. These features are then fused after processing of channel self-attention. Notably, unlike those of SE and CVBM, the proposed channel attention exploits dot product operation instead of global average pooling to yield the attention weights, which is shown in Figure 14.  
Atrous spatial pyramid pooling (ASPP) is often used in CNN to enlarge the receptive field. [PERSON] et al. fostered a multi-scale spatial attention-guided block for the task of self-supervised MDE [51]. The multi-scale spatial attention-guided block consists of ASPP modules followed by spatial attention constructed by parallel spatial self-attention and cross-attention similar to that of CBAM.