# Monocular Depth Estimation Using Res-UNet with an Attention Model
## 3 Proposed Method
### Proposed Model
#### 3.2.3 Attention Model in Depth Maps Estimation  
In order to reduce the loss and improve the accuracy, current frameworks rely on additional preceding object localization models. We show that incorporating an attention mechanism into a common CNN model can accomplish the same goal. This does not necessitate the training of many additional model parameters or numerous models. The attention mechanism gradually reduces feature responses in irrelevant background regions ([PERSON] et al., 2014). This process does not need to crop the region of interest (ROI) between networks, in contrast to the localization strategy in multi-stage CNNs. The coefficients of attention, \(\alpha\in[0,1]\) in Equation (3), identify key image regions and prune feature responses to retain only the activations that are relevant to the task at hand. Input feature maps and attention coefficients are multiplied element-wise by attention gates to produce the output (see Equation (4)). \[x_{i,c}^{l}=x_{i,c}^{l}\cdot\alpha \tag{4}\]  
Here, \(x^{l}\) is the feature map at the output of layer \(l\), \(i\) and \(c\) denote the spatial and channel dimensions, and \(\alpha\) is the attention weight coefficient. By default, each pixel vector \(x_{i,c}^{l}\in R^{F_{l}}\) is given a single scalar attention value, where \(F_{l}\) is the number of feature-maps in the layer \(l\). In our model we use multi-dimensional attention coefficients because there were several depth values (classes). We used additive attention to obtain gating coefficients following the same technique as [PERSON] et al. (2014). Despite requiring greater processing resources, this has been empirically proven to produce results that are more accurate than multiplicative attention. The formulation of additive attention is as follows. \[q_{att}^{l}=\psi^{T}\Big{(}\sigma_{1}\Big{(}W_{x}^{T}x_{i}^{l}+W_{g}^{T}g_{i}+ b_{g}\Big{)}\Big{)}+b_{\psi} \tag{5}\]  
\[\alpha_{i}^{l}=\sigma_{2}\Big{(}q_{att}^{l}\Big{(}x_{i}^{l},\ g_{i};\Phi_{att} \Big{)}\Big{)} \tag{6}\]  
Here, in Equations (5) and (6), \(\sigma_{2}\) denotes the sigmoid activation function, \(x_{i}^{l}\) and \(g_{i}\) are features from the encoder and decoder layers. The attention gate is characterized by a set of parameters, \(\Phi_{att}\), containing linear transformations: \(W_{x}\in R^{F_{l}xF_{int}}\), \(W_{g}\in R^{(F_{g}xF_{int})}\), \(\psi^{T}\in R^{(F_{int})}\), and bias terms: \(b_{\psi}\in R\), \(b_{g}\in R^{(F_{int})}\). \(R^{(F_{int})}\) represents dimensional intermediate space and \(R^{(F_{l})}\) is the dimensional feature-map layer. \(W_{x}\) represents the weights from the encoder layer, while \(W_{g}\) represents the weights generated from the gating signal from the decoder. \(\psi^{T}\) is the linear transformation of the output. Here, the linear transform is performed by convolving a \(1\times 1\times 1\) channel-wise operation to the input tensor. Such an operation is termed vector concatenation-based attention. A sigmoid activation, \(\sigma_{2}\), is used which results in better convergence for attention-gate parameters. The proposed attention mechanism is shown in Figure 6.  
In Figure 6a, at the gating, there are two inputs \(g\) and \(x\). The input \(g\) is coming from the decoder of the network, carrying better feature representation; \(x\) comes from the encoder through skip connections, meaning it has better spatial information. The job of attention is to pass those features which are more relevant to the output, such relevant features have higher weights while irrelevant features have lower weights. Attention is fed by both \(x\) and \(g\) at the same time. Attention applies a \(1\times 1\) convolution with a stride of \(1\) on input \(g\), while on input \(x\) it applies a convolution of \(1\times 1\) with the stride of \(2\). The convolution with strides of \(2\) on \(x\) resizes its dimensions to the size of \(g\). Thus, they are added with the same dimensions, and the operation can move forward to the next step.