# Depth-Adaptive Deep Neural Network Based on Learning Layer Relevance Weights
## 3 Depth-Adaptive Deep Neural Network
### Layer Pruning
The proposed ANNA approaches succeeded in guiding the learning process on the plain-18 network into a compact one-layer model. Specifically, class-independent and class-dependent ANNAs with plain-18, in one epoch, achieved an accuracy of 56.48% (17% higher than ResNet-18) and 56.14% (16% higher than ResNet-18), respectively. Furthermore, applying ANNA approaches on the plain-34 network resulted in a 2-layer model. The obtained models outperformed ResNet-34, with a difference in terms of accuracy of 9% for class-independent ANNA and 7% for class-dependent ANNA. In-order to investigate the effect of per-class accuracies on the overall performance, Figure 15a presents the per-class accuracies of ResNet in comparison with class-dependent ANNA applied on plain-18 and plain-34. As can be noticed, the achieved accuracies by the class-independent ANNA models with respect to classes \"cat\", \"deer\", and \"dog\" are significantly higher than the accuracies obtained by the ResNet models. This affects the overall performance. The ROC curves of the ResNets and class-independent ANNAs on the CIFAR-10 dataset [43], as displayed in Figure 15b, show consistent results with the obtained accuracies and F1-scores. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **Accuracy** & **Error** & **F1-Score** & **AUC** & **Depth** \\ \hline Plain-18 & 44.28 & 55.72 & 0.44 & 0.86 & 18 layers \\ ResNet-18 & 48.2 & 51.8 & 0.47 & 0.88 & 18 layers \\ Class-Independent ANNA + Plain-18 & **56.48** & **43.52** & **0.56** & **0.91** & **1 layer** \\ Class-Dependent ANNA + Plain-18 & 56.14 & 43.86 & 0.56 & 0.91 & 1 layer \\ \hline Plain-34 & 26 & 74 & 0.18 & 0.76 & 34 layers \\ ResNet-34 & 49.36 & 50.64 & 0.48 & 0.88 & 34 layers \\ Class-Independent ANNA + Plain-34 & **53.91** & **46.09** & **0.53** & **0.9** & **2 layers** \\ Class-Dependent ANNA + Plain-34 & 52.9 & 47.1 & 0.53 & 0.9 & 2 layers \\ \hline Class-Independent ANNA + Plain-50 & 47.71 & 52.29 & 0.45 & 0.87 & 2 layers \\ Class-Dependent ANNA + Plain-50 & **49.2** & **50.8** & **0.47** & **0.88** & **2 layers** \\ \hline \hline \end{tabular} Bold number indicates best performance.

\end{table} 
Table 4: Performance measure comparison on CIFAR-10 dataset. Figure 14: Comparison of ResNet and class-dependent ANNA. (**a**) Overall accuracy and per-class accuracies achieved by ResNet and class-dependent ANNA on MNIST dataset; (**b**) ResNet and class-dependent ANNA ROC curves on MNIST dataset. Table 5 compares the performance of the proposed ANNA approaches to Plain-18, Plain-34, ResNet-18, and ResNet-34 models in terms of testing time. As can be seen, the models learned by ANNAs are faster than the other considered models. This can be explained by the fact that ANNAs learned a compact model that had fewer layers than the other models. The depth effect is obvious when comparing the testing time of plain-34 and ResNet-34 to the testing time of models learned by ANNAs. For instance, ResNet-34 testing time on the MNIST dataset is double the testing time of the model obtained from applying ANNA on plain-34. As shown by the reported results in Figure 16, the learned layer weights increase as the depth increases. This provides an interpretation of the importance of the layers. In fact, it indicates that the deeper layers are more relevant than initial ones since they learn more complex and abstract features. Figure 15: Comparison of ResNet and class-independent ANNA. (**a**) Overall accuracy and per-class accuracies achieved by ResNet and class-independent ANNA on CIFAR-10 dataset; (**b**) ResNet and class-independent ANNA ROC curves on CIFAR-10 dataset.