# Depth-Adaptive Deep Neural Network Based on Learning Layer Relevance Weights
## 3 Depth-Adaptive Deep Neural Network
### Layer Pruning
Moreover, despite the good performance of AdaNet on the MNIST dataset [42], it has a large testing time in comparison with the ANNA models. This also applies for the CIFAR-10 dataset [43]. This can be explained by the light weight of ANNA models. In fact, the learned model architecture has fewer layers than the other models. ## 6 Conclusions

Designing DNNs that exhibit the appropriate level of complexity and depth is a challenging task. In this paper, we proposed two adaptive deep learning approaches intended to learn the optimal DNN depth. Moreover, both approaches assign fuzzy relevance weights to the network layers. In fact, the learned weights yield better interpretability of the model and the extracted features. Furthermore, the proposed approaches train the DNN model and learn the layers fuzzy relevance weights jointly. Specifically, the DNN architecture is updated by pruning the non-relevant layers. The learning process is formulated as an optimization of a novel cost function. The proposed approaches were assessed using standard datasets and performance measures. The experiments proved that the proposed approaches can automatically learn the optimal number of layers. Moreover, when compared to state-of-the-art approaches, ANNAs gave competitive results on MNIST and outperformed the other models on CFAR-10. Furthermore, it has been shown that the proposed approaches overtake the state-of-the-art approaches in terms of time efficiency and number of learnable parameters. As future works, we suggest research to learn the architecture of deep learning networks by incrementally adding layers instead of starting with an overestimated number, \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{3}{*}{**Model**} & \multirow{3}{*}{**FLOPs (G)**} & \multicolumn{2}{c}{**MNIST**} & \multicolumn{2}{c}{**CIFAR-10**} \\ \cline{3-7}  & & \multicolumn{2}{c}{**Time (s)**} & \multicolumn{2}{c}{**FLOPs**} & \multicolumn{2}{c}{**Time (s)**} \\ \cline{3-7}  & & **Training** & **Testing** & **(G)** & **Training** & **Testing** \\ \hline ResNet-18 & 0.0713 & 23.89 & 1.67 & 0.162 & 22.07 & 1.84 \\ ResNet-34 & 0.147 & 37.88 & 2.18 & 0.326 & 34.2 & 1.99 \\ \hline AdaNet & 0.00953 & 12.3 & 5.92 & 0.00761 & 11.58 & 5.63 \\ \hline DenseNet-121 & 0.102 & 57.5 & 4.12 & 0.117 & 50.43 & 4.16 \\ DenseNet-169 & 0.124 & 80.55 & 5.93 & 0.139 & 71.58 & 5.99 \\ \hline A\({}^{2}\)S\({}^{2}\)K-Resnet & 0.0984 & 15.09 & 1.17 & 0.1312 & 16.41 & 1.53 \\ \hline Class-Independent ANNA + Plain-18 & 0.00666 & 292.26 & 1.01 & 0.00695 & 201.94 & 1.02 \\ Class-Dependent ANNA + Plain-18 & 0.00666 & 263.29 & 0.99 & 0.00695 & 194.09 & 1.03 \\ Class-Independent ANNA + Plain-34 & 0.0114 & 523.73 & 1.06 & 0.0129 & 357 & 1.02 \\ Class-Dependent ANNA + Plain-34 & 0.0114 & 412.88 & 1.06 & 0.0129 & 366.15 & 1.02 \\ Class-Independent ANNA + Plain-50 & 0.0155 & 659.91 & 1.0 & 0.0214 & 556.11 & 1.01 \\ Class-Dependent ANNA + Plain-50 & 0.0155 & 664.83 & 1.01 & 0.0214 & 554.56 & 1.02 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Computational complexity and time comparison of ANNA and state-of-the-art approaches on MNIST dataset and CIFAR-10 dataset. and the, discarding the non-relevant layers while training the model. This would reduce the training time, since the experiments showed that networks with a small optimal number of layers perform better than deeper ones. Conceptualization, A.A., O.B. and M.M.B.I. ; methodology, A.A., O.B. and M.M.B.I. ; software, A.A.; validation, A.A. and O.B. ; formal analysis, A.A. and O.B. ; investigation, A.A. and O.B. ; resources, A.A. and O.B. ; data curation, A.A.; writing--original draft preparation, A.A.; writing--review and editing, A.A. and O.B. ; visualization, A.A. and O.B. ; supervision, O.B. and M.M.B.I. All authors have read and agreed to the published version of the manuscript. This research received no external funding. Not Applicable. **Data Availability Statement:** Publicly available datasets were analyzed in this study.