# Attention Mechanism Used in Monocular Depth Estimation: An Overview
## 3 Attention Mechanism in MDE Methods
### Transformer-Based Attention Methods  
#### 3.3.1 Supervised Training  
Dense prediction Transformer (DPT) is the first MDE model that uses ViT as a backbone [10]. The features of the different stages of the encoder, extracted by Transformer blocks, are resampled and fused by convolutional operations followed by the prediction head to the output depth map. Compared with the state-of-the-art CNN-based method, DPT gains a remarkable 28% improvement in terms of accuracy. However, the DPT needs a large amount of data to be trained because convolutional operations are entirely removed in the encoder (DPT-base and DPT-large). The overview of DPT model is shown in Figure 16.  
DPT inspires a surge of investigations of Transformer for MDE [15; 59], as well as applications of downstream tasks [60]. [PERSON] et al. proposed a rich global feature-guided network based on Transformer [61], which highlights global information fusion in the decoder stage by large kernel convolution attention. Most existing MDE methods utilize a regression strategy to predict depth; however, BinsFormer [16] formulates it as a regression classification problem by predicting the bins center and the probability that a certain pixels belong to certain bins. In their scheme, Transformer is used to adaptively predict bins, while any existing backbone can be used to extract features. We categorize BinsFormer as a kind of Transformer-based method since the performance is better when Transformer is  
Figure 15: Structure of TSAM (image: courtesy to [PERSON] et al. [56]).  
used as the backbone. Depthformer also adopts a regression classification strategy that uses a Transformer module to predict the bins and an encoder that is the same as that of DPT to extract features [17].  
Transformer suffers heavy computation since the multi-head spatial attention operations are conducted over the holistic image. To alleviate the computational payload, SW-Transformer limits the attention operations into specific windows [62]. [PERSON] et al. proposed a MDE method using SW-Transformer as the backbone [63]; meanwhile, CBAMs are used in the decoder stage to fuse multi-scale features. [PERSON] et al. also adopted SW-Transformer to extract intermediate features [11]. In addition, the commonly used skip connection is replaced by a skip attention module, which fuses lower-resolution features of the decoder and higher-resolution features of the encoder in a cross-attention way. In line with the goal of computational reduction, RA-Swin [64] was proposed for MDE. RA-Swin applies SW-Transformer to extract hierarchical features followed by Refine Net fusing the multi-scale features in the decoder stage. An alternative strategy for mitigating the computational payload of Transformer is to simplify the architecture of Transformer. [PERSON] et al. reported a lightweight MDE framework in which the number of Transformer blocks is reduced from 12 to 6 or 4 [65]. [PERSON] et al. proposed a real-time MDE method named SideRT using Swin Transformer as the backbone [66]. The contribution of SideRT is that cross-attentions between adjacent scales of feature maps are conducted to capture global information.