# Depth-Adaptive Deep Neural Network Based on Learning Layer Relevance Weights
## 3 Depth-Adaptive Deep Neural Network
### Layer Pruning
(**a**) Accuracy achieved by the plain-50 network with respect to different depths using CIFAR-10 dataset; (**b**) Per-class accuracies achieved by the plain-50 network with respect to different depths using CIFAR-10 dataset. Figure 10 reports the overall accuracy and the per-class accuracies obtained using the class-independent and the class-dependent ANNAs when using the MNIST dataset [42]. As can be seen, similar performance is attained with respect to all digit categories [42]. Moreover, both ANNA approaches yield comparable classification performance. This proves that the layer relevance models, either class-dependent or class-independent, guided the learning process towards the optimal network depth dynamically while training the model. Furthermore, the obtained layer fuzzy relevance weights allow a straightforward interpretation of the network behavior. In other words, they reflect the importance of the different network layers and their impact on the discrimination power of the learned models. Figure 8: Layer relevance weights learned by the proposed class-independent ANNA on MNIST dataset after training the model using: (**a**) 1 batch; (**b**) 4 batches; (**c**) 6 batches; (**d**) 15 batches. Figure 9: Layer relevance weights learned by the proposed class-dependent ANNA on MNIST dataset after training the model using: (**a**) 1 batch; (**b**) 4 batches; (**c**) 6 batches; (**d**) 15 batches. Figures 11 and 12 show sample weights learned by class-independent and class-dependent ANNA, respectively, when using the CIFAR-10 dataset [43]. These sample weights are obtained after training the model using 1, 5, 6, and 7 batches. Both proposed approaches converge to one hidden-layer network. This is in accordance with the findings of the empirical experimentation conducted in experiment 1. Moreover, the proposed class-independent and class-dependent ANNAs yield lightweight models, since the number of learnable parameters decreased from 790,986 to 337,834. The class-independent and class-dependent ANNAs achieved comparable time performance, with 556.116 and 554.564 s, respectively, to complete the training process. Evidently, learning the network depth using one of the proposed ANNA approaches is more efficient than empirical investigation of the optimal depth, where 6720 s was required to conduct the whole experiment, aiming at determining manually the optimal network depth. Figure 11: Layers’ relevance weights learned by the proposed class-independent ANNA on CIFAR-10 dataset after training the model using: (a) 1 batch; (b) 5 batches; (c) 6 batches; (d) 7 batches. Figure 10: Overall accuracy and per-class accuracies obtained using the proposed ANNA approaches and MNIST dataset. The overall accuracy and the per-class accuracies obtained using the class-independent and the class-dependent ANNAs when using CIFAR-10 dataset [43] are reported in Figure 13. As can be seen, the performance varies with respect to the image categories. For instance, the models achieved an accuracy of 10% and 30% for \"cat\" and \"truck\" classes, respectively, while attaining an accuracy of 79% for the \"automobile\" class. As mentioned in experiment 1, the accuracy obtained for class \"cat\" improves as the network depth increases. This explains the poor performance of the one hidden-layer network for class \"cat\". This affected the overall accuracy of both ANNA approaches. Class-independent and class-dependent ANNA approaches achieved comparable classification performances. To be more accurate, for the CIFAR-10 dataset [43], the class-dependent ANNA approach achieved a slightly better performance than the class-independent ANNA. However, both the ANNA approaches succeeded in determining an appropriate network depth and enhancing the overall classification performance through the learning process. Figure 12: Layers’ relevance weights learned by the proposed class-dependent ANNA on CIFAR-10 dataset after training the model using: (**a**) 1 batch; (**b**) 5 batches; (**c**) 6 batches; (**d**) 7 batches.