# Depth-Adaptive Deep Neural Network Based on Learning Layer Relevance Weights
## 3 Depth-Adaptive Deep Neural Network
### Layer Pruning
Based on these experimental results, we can conclude that a single hidden-layer model yields the best results for the considered dataset, and outperforms the other deeper plain models. The previously mentioned experiments conducted on the MNIST [42] dataset were conducted on CIFAR-10 [43] dataset. The accuracies achieved using CIFAR-10 [43] are shown in Figure 7a. The obtained results affirm the findings of the MNIST [42] experiment. Specifically, the accuracy decreases as the network depth increases. Moreover, the results show that the networks of depths between 1 and 3 achieved better performance than deeper models. As can be seen in Figure 7b, except for class \"cat\", all other classes achieved the maximum accuracy when using 1 to 3-layer-depth networks. Moreover, adding one layer to a 7-layer-depth network-ork causes a slight decrease in the performance. Furthermore, 8 to 50-layer-depth models exhibit instability in their performance. As for class \"cat\", the performance suffers from fluctuations as the depth changes. However, the general accuracy of this category enhances as the depth increases. This behavior explains the overall accuracy drop for the CIFAR-10 dataset [43]. Figure 6: MNIST dataset performance results. (**a**) Accuracy achieved by the plain-50 network with respect to different depths using the MNIST dataset; (**b**) Per-class accuracies achieved by the plain-50 network with respect to different depths using the MNIST dataset. From the experiment results reported in Figures 6a and 7a, we can confirm the counterintuitive finding stating that deeper networks lead to lower accuracy. This known as the degradation problem, which residual networks aim to solve. ### Experiment 2: Performance Assessment of ANNA Approaches Applied to Plain Networks

Figure 8 exhibits sample layer relevance weights learned by the proposed class-independent ANNA on the MNIST dataset [42] after training the model using 1, 4, 6, and 15 batches. As can be seen, after training the model using one batch, the maximum weight is assigned to layer 46. Therefore, according to the proposed pruning approach, all hidden layers from 47 to 50 are pruned, which results in a 46 hidden-layer network. Similarly, after training the model using 4 batches, layer 27 is assigned the maximum weight. Hence, all hidden layers from 28 to 46 are pruned, which results in a 27 hidden-layer network. In the same way, after training the model using 6 batches, layers from 23 to 27 are pruned, which yields a 22 hidden-layer network. After 15 batches are conveyed to the network, layers from 9 to 22 are pruned, which results in 8 hidden-layer network. At the end, the learning process converges to a one hidden-layer network when the model is trained using 41 batches. Similarly, Figure 9 depicts sample layer weights learned using the class-dependent ANNA on the MNIST dataset [42] after training the model using 1, 4, 6, and 15 batches. As can be seen, after training the model using the first batch, the last four layers are pruned, since layer 46 has the largest weight. After using 4 batches, layer 27 is assigned the largest weight, thus all subsequent layers are pruned. Layer 22 becomes the most relevant layer after batch 6, hence layers from 23 to 27 are pruned. In the same way, the highest relevance weight is assigned to layer 8 after 15 batches are conveyed to the network. Therefore, layers 9 to 22 are pruned. Later on, when using 9 batches, the model converges to one single hidden-layer network. As such, the number of learnable parameters decreases from 713,610, at the beginning of the training process, to 260,458 when the proposed approach converges. In fact, the proposed class-dependent and class-independent ANNA approaches yield lightweight models, reduce the overfitting risk, and improve the model generalization. Figure 7: CIFAR-10 dataset performance results.