# Depth-Adaptive Deep Neural Network Based on Learning Layer Relevance Weights
## 3 Depth-Adaptive Deep Neural Network
### Layer Pruning
Figure 5:0 Overall accuracy and per-class accuracies obtained using the proposed ANNA approaches when using CIFAR-10 dataset. ### Experiment 3: Performance Assessment of ANNA Approaches Applied to ResNet Plain Models

Table 3 reports the achieved performances on the MNIST dataset [42] in terms of accuracy, error rate, F1-measure, and area under curve (AUC). Specifically, it compares the results of the application of ANNAs to plain-18 and plain-34, to the results of plain-18, plain-34, ResNet-18, ResNet-34, and plain-50 CNN networks. As can be seen, the proposed ANNA approaches succeed in guiding the learning process into a lightweight version of plain-18 and plain-34. More specifically, it converges to 2-layer network in the case of plain-18 and a 3-layer network in the case of plain-34. Moreover, the learned lightweight models outperformed the plain baseline networks and ResNet models. In fact, the 2-convolutional layer network learned by the class-dependent ANNA using the plain-18 model achieved the highest accuracy of 89.9%. As depicted in Table 3, the accuracy achieved by the plain-34 baseline network did not exceed 14.48%. This is an expected result due to the degradation problem, which affects networks with high depth. In fact, ResNet models tackled this problem through the use of skip connections. As can be seen from Table 3, ResNet does perform better than the plain model. Nevertheless, the class-independent and class-dependent ANNAs were able to better overcome the plain-34 network degradation problem by achieving an accuracy of 86.18% and 87.97%, respectively. Similar results can be seen in the performances of plain-18, ResNet-18 and ANNAs approaches. Therefore, the compact models learned by ANNAs proposed approaches outperformed the ResNet-18 and ResNet-34 models. Comparing these results with the performance of applying ANNAs on the 50-layer CNN model shows that a simple network design with an optimal depth learned by the proposed ANNAs approaches outperforms plain-18, ResNet-18, plain-34, and ResNet-34. The analysis of these results based on the accuracy performance holds for the other performances depicted in Table 3, namely the error rate, the F1-score, and AUC. In fact, the considered performance measures are consistent with each other. Figure 13(a) compares the per-class accuracies of ResNet models with class-dependent ANNA applied on plain-18 and plain-34. As can be seen, for categories \"1\", \"2\", and \"3\", the class-dependent ANNA models outperformed ResNet-18 (and ResNet-34 for category \"3\") by a significant margin. For the remaining categories, the class-dependent ANNA models presented a competitive or better performance than ResNet models. Figure 13(b) shows the receiver operating characteristic curve (ROC) for the ResNet models and class-dependent ANNA models. As exhibited by Figure 13, the proposed ANNAs approaches perform significantly better than ResNet models on the MNIST dataset [42]. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **Accuracy** & **Error** & **FI-Score** & **AUC** & **Depth** \\ \hline Plain-18 & 61.04 & 38.96 & 0.55 & 0.85 & 18 layers \\ ResNet-18 & 61.67 & 38.33 & 0.59 & 0.87 & 18 layers \\ Class-Independent ANNA + Plain-18 & 87.82 & 12.18 & 0.87 & 0.98 & 2 layers \\ Class-Dependent ANNA + Plain-18 & **89.9** & **10.1** & **0.9** & **0.99** & **2 layers** \\ \hline Plain-34 & 14.48 & 85.52 & 0.07 & 0.5 & 34 layers \\ ResNet-34 & 46.59 & 53.41 & 0.4 & 0.77 & 34 layers \\ Class-Independent ANNA + Plain-34 & 86.18 & 13.82 & 0.86 & 0.98 & 3 layers \\ Class-Dependent ANNA + Plain-34 & **87.97** & **12.03** & **0.88** & **0.98** & **3 layers** \\ \hline Class-Independent ANNA + Plain-50 & **87.58** & **12.42** & **0.88** & **0.98** & **2 layers** \\ Class-Dependent ANNA + Plain-50 & 86.61 & 13.39 & 0.87 & 0.98 & 2 layers \\ \hline \hline \end{tabular} Bold number indicates best performance.

\end{table} 
Table 3: Performance measure comparison on MNIST dataset. Table 4 shows the achieved accuracies on the CIFAR-10 dataset [43].