# Depth-Adaptive Deep Neural Network Based on Learning Layer Relevance Weights
## 3 Depth-Adaptive Deep Neural Network  
We propose to automatically learn the optimal DNN depth along with fuzzy relevance weights for each DNN layer. Particularly, we introduce a technique to simultaneously learn the network depth and train the deep neural network model. A novel cost function that optimizes the relevance weight of each layer along with the model hyperparameter is designed to formulate the learning procedure. In other words, we propose to prune the non-relevant layers and accordingly update the deep neural network architecture. As a result, two Adaptive Neural Network Adaptive approaches (ANNAs) are proposed to learn the relevance of each layer, which alleviates both the depth setting problem and the interpretability problem. Specifically, the first ANNA approach is class independent, and the learned layer fuzzy relevance weights do not depend on the pre-defined classes. On the other hand, the second ANNA approach learns layer fuzzy relevance weights that depend on the classes set for the classification problem. One should note that the proposed approaches are generic and applicable to any supervised deep neural network, regardless of its activation function and/or regularization technique.  
In particular, as can be seen in Figure 1, the network has \(\mathcal{L}\) layers, where the \(\ell^{th}\) layer deploys a non-linear transformation \(\mathcal{F}\) on the input \(x_{j}\) to produce the output \(y_{j}\) using the following:  
\[\hat{y_{j}}=\mathcal{F}\Big{(}x_{j};\mathcal{W}^{\mathcal{F}}\Big{)} \tag{6}\]  
where \(\big{\{}x_{j}\big{\}}_{j=1..N}\) is a set of \(N\) instances fed as input to the deep learning algorithm, and \(\mathcal{W}^{\mathcal{F}}\) represents the set of network parameters (neuron weights). Depending on \(\mathcal{F}\), the feedforward network can be a Fully Connected Network (FCN) or a Convolutional Neural Network (CNN).  
As shown in Figure 1, we introduce a Layer Relevance Weight Learning component. It aims at learning iteratively fuzzy relevance weights for the network layers. For this purpose, we define the distance \(d^{\prime}_{jk}\) between the input vector and its mapped vector \(x^{f}_{j}\) at  
Figure 1: The block diagram of the proposed system.  
layer \(\ell\). The distance \(d^{\ell}_{jk}\) represents the degree to which two instances \(x_{j}\) and \(x_{k}\) are related according to their representations at layer \(\ell\). This distance is obtained using the Euclidian distance as follows:  
\[d^{\ell}_{jk}=\left\|x^{\ell}_{j}-x^{\ell}_{k}\right\|_{2} \tag{7}\]  
We also define the membership coefficient \(\gamma_{ij}\) as  
\[\begin{cases}\gamma_{ij}=1,\ \ if\ j\in c_{i}\\ \gamma_{ij}=0,\ \ if\ j\
otin c_{i}\end{cases} \tag{8}\]  
Note that each input instance \(x_{j}\) belongs to a given class \(c_{i}\).  
In this paper, we introduce two approaches to learn fuzzy relevance weights with respect to each layer \(\ell\). These weights reflect the contribution of each layer in the classification task. In fact, a small fuzzy weight indicates that the layer does not contribute sufficiently and therefore can be discarded. Alternatively, a large fuzzy weight indicates that the corresponding layer contributes heavily in the classification and therefore should be maintained. For this purpose, two approaches are proposed. The first ANNA approach is class independent. In other words, the fuzzy weights are learned globally for all classes, and they do not depend on the pre-defined classes. On the other hand, the second ANNA approach learns fuzzy weights with respect to the problem classes.