# LUMDE: Light-Weight Unsupervised Monocular Depth Estimation via Knowledge Distillation
## 1 Introduction  
3D geometric reconstruction from 2D images has been an important research direction in computer vision. The acquisition of depth information is also a significant technical node in some practical applications such as robots, automatic driving vehicles, virtual reality, augmented reality, etc. Most depth estimation algorithms are based on supervised neural networks. For example, [PERSON] et al. [1] first reported a deep-learning-based method for monocular depth estimation using a multi-scale convolutional neural network (CNN). Later studies introduce better network architecture [2], or more sophisticated training loss functions [3, 4, 5], to improve performance. In addition, several methods [6, 7] use two networks, one for depth estimation and the other for motion, to mimic the structure from motion (SfM), or simultaneous localization and mapping (SLAM) in supervised training. As supervised learning relies on ground truth data, which has to be acquired from extra, and usually more expensive, sensors, such as Lidar or an RGBD camera, the unsupervised depth estimation network approach has seen rapid progress in recent years. An unsupervised depth network can take data from either binocular images or sequential monocular images as input, with the latter being of more practical interest, as monocular cameras are already readily available in most autonomous devices or vehicles. For monocular depth estimation, the pioneering work is from [PERSON] et al. [8], who employed a depth estimation network along with a pose network to construct the photometric loss between consecutive temporal frames. Following their work, many subsequent methods have triedto improve the performance of self-supervised depth estimation, such as Monodepth2 [9] and SC-SfMLearner [10]. Specifically, Monodepth2 employs per-pixel minimum photometric loss, the auto-masking idea, and multi-scale depth estimation to significantly improve the performance of self-supervised depth prediction; while SC-SfMLearner proposes scale consistency to constrain depth estimation and a weight mask to address dynamic objects. However, with the continuous improvement in model accuracy, the parameters of the models range from tens of millions to more than 100 M, which is computationally expensive for resource-limited and low-cost edge devices. Therefore, considering the memory, computational capability, and power consumption of devices, as well as the needs of real-time applications, model compression becomes essential. The prevailing model compression methods include quantization [11], pruning [12], parameter sharing, and knowledge distillation [13]. As an unsupervised depth estimation distillation task, [PERSON] et al. [14] distill information between three computer vision (CV) tasks, i.e., monocular-depth, semantic-segmentation, and surface-normal. This cross-task distillation focuses on the collaborative training between tasks, rather than the compression of parameters and computation. To the best of our knowledge, little work is performed on unsupervised depth model distillation. Inspired by [PERSON] et al. [15], we use the strategy of knowledge distillation, which contains pixel-wise loss, pair-wise loss, and holistic loss, to distill depth information from the unsupervised teacher model, thereby generating a lightweight student model in this paper. Due to the lack of ground truth and considering the limitation of unsupervised learning, we propose two assisted modules to further improve the performance of the student depth network. One is a pose network, which is widely used in unsupervised depth models, the other is Transformer [16], which can better capture holistic information compared to CNN, to improve the performance of the discriminator of the Generative Adversarial Network (GAN). Our aim is to compress the student network while maintaining reasonable accuracy.