# Depth-Adaptive Deep Neural Network Based on Learning Layer Relevance Weights
## 3 Depth-Adaptive Deep Neural Network
### Layer Pruning
\begin{table}
\begin{tabular}{c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{2}{c}{**Testing Time (s)**} \\ \cline{2-3}  & **MNIST** & **CIFAR-10** \\ \hline Plain-18 & 1.61 & 1.71 \\ ResNet-18 & 1.67 & 1.84 \\ Class-Independent ANNA + Plain-18 & 1.01 & 1.02 \\ Class-Dependent ANNA + Plain-18 & 0.99 & 1.03 \\ \hline Plain-34 & 2.18 & 2.04 \\ ResNet-34 & 2.18 & 1.99 \\ Class-Independent ANNA + Plain-34 & 1.06 & 1.02 \\ Class-Dependent ANNA + Plain-34 & 1.06 & 1.02 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Testing time on MNIST dataset and CIFAR-10 dataset. ### Experiment 4: Performance Comparison of ANNA Approaches with State-of-the-Art Models

Table 6 reports the performance of all considered models. As can be seen, on the MNIST dataset [42], AdaNet models achieved the highest performances with an accuracy equal to 92.45%. Nevertheless, ANNA models gave competitive results, with an accuracy of 89.9% for class-dependent ANNA applied to plain-18, and around 87% for the other models. Moreover, the proposed approaches outperformed ResNet, DenseNet, and A\({}^{2}\)S\({}^{2}\)K-ResNet models for MNIST data. As for the CIFAR-10 dataset [43], ANNA approaches applied to plain-18 surpassed the accuracy of the state-of-the-art approaches by at least by 13%. In order to compare the computational complexity of the proposed ANNA approaches with state-of-the-art methods, the number of floating-point operations (FLOPs) were calculated for each model. As it can be seen in Table 7, the compact models of the ANNA approaches have lower complexity than the state-of-the-art models. Particularly, the model resulting from applying ANNA approaches to plain-18 achieved the least number of FLOPs (0.00666) and a high accuracy of 89.9% on the MNIST dataset. Similarly, they achieved the highest accuracy with the lowest FLOPs (0.00695) on CIFAR-10. Thus, the \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{2}{c}{**MNIST**} & \multicolumn{2}{c}{**CIFAR-10**} \\ \cline{2-5}  & **Accuracy** & **Error** & **Accuracy** & **Error** \\ \hline ResNet-18 & 48.2 & 51.8 & 48.2 & 51.8 \\ ResNet-34 & 46.59 & 53.41 & 49.36 & 50.64 \\ \hline AdaNet & **92.45** & **7.55** & 15.7 & 84.3 \\ \hline DenseNet-121 & 24.5 & 75.5 & 49.86 & 50.14 \\ DenseNet-169 & 19.1 & 80.9 & 42.86 & 57.14 \\ \hline A\({}^{2}\)S\({}^{2}\)K-Resnet & 40.78 & 59.22 & 31.04 & 68.96 \\ \hline Class-Independent ANNA + Plain-18 & 87.82 & 12.18 & **56.48** & **43.52** \\ Class-Dependent ANNA + Plain-18 & **89.9** & **10.1** & **56.14** & **43.86** \\ Class-Independent ANNA + Plain-34 & 86.18 & 13.82 & 53.91 & 46.09 \\ Class-Dependent ANNA + Plain-34 & 87.97 & 12.03 & 52.9 & 47.1 \\ Class-Independent ANNA + Plain-50 & 87.58 & 12.42 & 47.71 & 52.29 \\ Class-Dependent ANNA + Plain-50 & 86.61 & 13.39 & 49.2 & 50.8 \\ \hline \hline \end{tabular} Bold number indicates best performance.

\end{table} 
Table 6: Performance measure comparison of ANNA and state-of-the-art approaches on MNIST dataset and CIFAR-10 dataset. Figure 16: Layers’ relevance weights learned by ANNA approaches when applied to Plain-18 and Plain-34 on MNIST and CIFAR-10 datasets. proposed approach succeeded in learning lightweight models with high performance and low complexity. Concerning the training time, as shown in Table 7, ANNA approaches exhibit large training time compared to the state-of-the-art approaches. This can be explained by the fact that the proposed approaches start with an over-estimation of the number of layers and then discard the non-relevant ones while training the model, whereas the learned optimal number of layers on the considered datasets is between 1 to 3. Nevertheless, the training is done offline; thus, it is not significant for real-time applications that use the already trained model. Regarding the testing time, ANNAs approaches outperform the considered state-of-the-art approaches. For instance, the DenseNet-121 testing time (on both datasets) is four times (at least) larger than all ANNA model testing times.