# LUMDE: Light-Weight Unsupervised Monocular Depth Estimation via Knowledge Distillation
## 4 Experiments
### Selection of Teacher and Student Network  
Both SC-SfMLearner [10] and Monodepth2 [9] were evaluated in this study to determine the most suitable teacher network. The relative performance of the two models is presented in Table 1. It can be seen that SC-SfMLearner has a better depth performance than Monodepth2. The backbone of SC-SfMLearner is ResNet18 [48], and the parameters are updated from ImageNet pre-trained weights. For the student network, several widely used lightweight backbones were tested on our baseline to explore the best student module, including the MobileNet series [49; 50; 51] and ShuffleNet series [52]. Our selection criterion is the trade-off between model size and accuracy. The experiments were performed on the baseline model by changing the backbone of the student DepthNet. The depth estimation performance was quantitatively evaluated with AbsRel, Log10, and RMS errors, as shown in Table 1. It was found that some student backbones fail to converge. MobileNet V3 (small) [51] outperforms the other backbones, probably due to the fact that it combines depth-wise separable convolution in MobileNet \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Type**} & \multirow{2}{*}{**Method**} & \multirow{2}{*}{**Pretrain**} & \multicolumn{2}{c}{**Error \(\downarrow\)**} & \multirow{2}{*}{**Model Size**} & \multirow{2}{*}{**Complexity**} \\ \cline{3-3} \cline{5-6}  & & & **AbsRel** & **Log10** & & & **RMS** \\ \hline \multirow{3}{*}{Teacher Network} & Monodepth2 [9] & \(\surd\) & 0.156 & 0.066 & 0.561 & **14.84 M** & **5.36 GMac/10.7 GFlops** \\  & & \(\times\) & 0.181 & 0.075 & 0.637 & **14.84 M** & **5.36 GMac/10.7 GFlops** \\  & SC-SfMLearner [10] & \(\surd\) & **0.148** & **0.062** & **0.545** & **14.84 M** & **5.36 GMac/10.7 GFlops** \\  & & \(\times\) & 0.170 & 0.072 & 0.603 & **14.84 M** & **5.36 GMac/10.7 GFlops** \\ \hline \multirow{6}{*}{Student Network} & MobileNet V1 [49] & \(\times\) & \multicolumn{2}{c}{No Convergence} & \multirow{6}{*}{3.62M} & None \\  & MobileNet V2 [50] & \(\surd\) & 0.169 & 0.071 & 0.589 & \multirow{6}{*}{3.62M} & \multirow{6}{*}{688.7 GMac/1.35 GFlops} \\  & & \(\times\) & 0.194 & 0.080 & 0.660 & & \\ \cline{1-1}  & MobileNet V3 (small) [51] & \(\surd\) & **0.163** & **0.068** & **0.581** & & \\ \cline{1-1}  & & \(\times\) & 0.186 & 0.077 & 0.639 & & **3.38 M** & **255.59 MMA/507.26 MFlops** \\ \cline{1-1}  & & \(\surd\) & \multicolumn{2}{c}{No Convergence} & \multirow{6}{*}{6.33 M} & \multirow{6}{*}{620 MMac/1.21 GFlops} \\ \cline{1-1}  & & \(\times\) & 0.183 & 0.076 & & & \\ \cline{1-1}  & ShuffleNet V2 (0.5) [52] & \(\surd\) & \multicolumn{2}{c}{No Convergence} & \multirow{6}{*}{4.03 M} & \multirow{6}{*}{790 MMac/1.55 GFlops} \\ \cline{1-1}  & ShuffleNet V2 (1.0) [52] & \(\surd\) & \multicolumn{2}{c}{No Convergence} & \multirow{6}{*}{5.79 M} & \multirow{6}{*}{1.15 GMac/2.30 GFlops} \\ \cline{1-1} \cline{5-6} \end{tabular} Note: The lowest error, parameter size, and computational complexity are marked in bold. Pretrain denotes adopting ImageNet pre-trained weights. The model size of the DepthNet includes the backbone encoder and decoder. M denotes a million parameters. Mac and Flops both denote the floating-point computation.  
\end{table} 
Table 1: Selection of teacher and student network. V1 [49] and inverted residuals, and the linear bottleneck in MobileNet V2 [50], and network configuration and parameters are explored by utilizing Neural Architecture Search (NAS) [53]. The size and complexity of the converged models are also summarized in Table 1. We found that MobileNet V3 (small) [51] has the smallest model size and best accuracy, so this backbone was adopted in the current LUMDE model.