# LUMDE: Light-Weight Unsupervised Monocular Depth Estimation via Knowledge Distillation
## 3 Method
### Preliminaries  
The current Knowledge Distillation Network is based on [15], which deals with dense prediction CV tasks such as semantic segmentation, object detection, and depth estimation. We adopt the ideas of pixel-wise loss, pair-wise loss, and holistic loss contained in the previous work. Pixel-Wise Distillation. In the original paper [15], the task is semantic segmentation, which outputs the probability distribution of different classes for the teacher and student networks. The loss design in the previous paper is not suitable for estimating depth that is  
Figure 1: Overview of the LUMDE architecture. Baseline results are generated by the knowledge distillation module with pixel-wise loss, pair-wise loss, and holistic loss (CNN discriminator). PoseNet is further incorporated into the student network to improve model performance during training, and Transformer is adopted to replace CNN as the discriminator in the GAN. spatially continuous, so, in this paper, we calculate the difference between the depth maps from the teacher and student networks with the loss function defined as follows:  
\[L_{pl}(S)=\frac{\sum\limits_{i\in R}\left(d_{i}^{t}-d_{i}^{s}\right)^{2}}{W\times H} \tag{1}\]  
where \(R\) represents all the depth map pixels, \(d_{i}^{t}\) and \(d_{i}^{s}\) denote the depth value of the \(i\)th pixel from the teacher network and student network, respectively, \(W\) and \(H\) denote the width and height of the depth map, respectively, and \(S\) indicates that the loss is used to update the parameters of the student module. The pixel-wise loss term is shown as \(L_{pi}\) in Figure 1.  
Pair-Wise Distillation. In addition to the straightforward difference between the depth maps at the pixel level, pair-wise loss is also applied in our model. This loss pays more attention to the structural similarity between intermediate feature maps produced by the teacher and student networks. In this part, two hyperparameters, i.e., connection range \(\alpha\) and granularity \(\beta\), are defined to represent the range on the maps we used to calculate the structure. Assuming the dimensions of the feature map are \(W^{\prime}\times H^{\prime}\times C\), the feature map can be transformed to an affinity graph for further comparison. Each pixel on the affinity map is aggregated by \(\beta\) pixels in the spatial local patch of the feature map, and we only consider structural similarity on the top-\(\alpha\) close pixels, i.e., the affinity map contains \(W^{\prime}\times H^{\prime}/\beta\) pixels and \((W^{\prime}\times H^{\prime}/\beta)\times\alpha\) range connections. The aggregation method is average pooling. If we assume \(f_{i}\) (dimension is \(1\times C\)) represents the \(i\)th pixel of the affinity map, \(a_{ij}^{t}\) and \(a_{ij}^{s}\) represent the structural similarity between the \(i\)th pixel and \(j\)th pixel of the teacher module and student module, respectively. The functions used to describe the discrepancy between the feature maps in the two modules can be defined as follows:  
\[a_{ij}=\frac{f_{i}^{T}f_{j}}{(\|f_{i}\|_{2}\times\|f_{j}\|_{2})} \tag{2}\]  
\[L_{pi}(S)=\frac{\beta}{W^{\prime}\times H^{\prime}\times\alpha}\sum\limits_{i \in R^{\prime}}\sum\limits_{j\in\alpha}\left(a_{ij}^{s}-a_{ij}^{t}\right)^{2} \tag{3}\]  
where \(R^{\prime}\) denotes the pixel set of the affinity graph. According to the experiment in [15], \(\beta=2\times 2\) and \(\alpha=W^{\prime}\times H^{\prime}/\beta\). \(S\) indicates that the loss is used to update the student module parameters. The pair-wise loss term is shown as \(L_{pi}\) in Figure 1.  
Holistic Distillation. Another strategy that we integrate into our framework is holistic distillation. This part can map the depth graphs generated from the teacher and student modules to high-order space and compute their holistic loss. Specifically, the original paper [15] employs a traditional self-attention CNN as a discriminator of a GAN and treats the outputs of the student and teacher net as fake and real samples, respectively.