# Attention Mechanism Used in Monocular Depth Estimation: An Overview
## 1 Introduction  
Depth information between the camera and object is crucial for applications such as 3D scene reconstruction, robotic avoidance, extended reality [1; 2], etc. Traditionally, depth information is obtained using technology such as LiDAR, stereo cameras, time of flight (TOF), etc. Recently, monocular depth estimation utilizing a single image attracted lots of interest because MDE is superior to other technologies in terms of cost, payload and ease of assembly. However, depth estimation from a single image is an ill-posed problem because a pixel of image corresponds to infinite physical points in the real world. To address this issue, researchers are inspired by the human vision, in which visual cues are exploited as prior knowledge to determine distances between objects, and even only one eye is used. Visual cues include features of object scale, shadow, occlusion, texture and shape, etc. Therefore, MDE can be formulated as an issue of uncovering the underlying non-linear mapping between visual cues and the corresponding depth information. In 2014, [PERSON] et al. proposed a supervised MDE method using CNN from only a single image [3]. Inspired by their groundbreaking work, tremendous MDE approaches are intensively investigated [4; 5; 6; 7]. An encoder-decoder architecture consisting of CNN is predominant in the community of MDE tasks. CNN uses convolutional operations within a local receptive field and downsampling to extract hierarchical features from input images [8; 9]. The lower layers have high resolution and a small receptive field, while the higher layers have low resolution and a large receptive field. [PERSON] et al. employ CNN for classification tasks, thereby resolving several issues with human development, such as education, economy, and epidemics [8]. In addition, [PERSON] et al. effectively detect the novel coronavirus from X-rays using CNN [9]. Different from image classificationand detection, dense prediction tasks, such as depth estimation, require a long-range context and detailed surroundings to determine the depth value of the central pixel [10]. However, the higher feature extracted by CNN loses too much detail. This situation is not fully addressed, though skip connections are utilized to supplement details from lower layers [11]. This drawback will lead to missed detection of small objects, edge blur, and incomplete images of objects. Therefore, CNN is not suitable for the MDE task [10]. To address the issue of the local receptive field of CNN, the attention mechanism is introduced into MDE to capture the long-distance dependency. Attention is inspired by the biological vision system and it is originally proposed to model the correlations between the source and the target in the sequence-to-sequence tasks [12]. Essentially, attention can reweight the information of inputs and yield a new representation with the global receptive field. In the MDE task, attention is conducted either in the spatial domain or the channel domain of the feature maps extracted by encoders. Experiments show that the points of the new representation that have more contributions are assigned higher weights, which leads to a remarkable improvement for MDE tasks [12]. Recently, Transformer was constructed using the multi-head self-attention modules in which convolutional operations are entirely discarded [13]. The Transformer gained significant success in the task of natural language processing so far. Furthermore, Transformer was adapted for computer vision tasks, thereby the vision Transformer (ViT) was proposed by [PERSON] et al. [14]. In 2021, [PERSON] et al. proposed the dense prediction Transformer (DPT), which uses the ViT as backbone architecture [10]. Leveraging the global receptive field and non-downsampling operations of the ViT, DPT outperforms the state-of-the-art CNN-based methods by a large margin in terms of accuracy for MDE tasks. After that, tremendous xFormers following this idea emerged in the community of MDE [11; 15; 16; 17].