# Depth-Adaptive Deep Neural Network Based on Learning Layer Relevance Weights
## 3 Depth-Adaptive Deep Neural Network
### Layer Pruning  
After the learning process of the layer fuzzy relevance weights terminates, the layer pruning component is initiated in order to drop the non-relevant layers. The layer pruning concerns the intermediate layers. In other words, the input and output layers are excluded. The layer pruner algorithm finds the layer with the maximum weight and prunes all following layers because increasing the network depth would yield poor representation. This indicates that the representation is enhanced by increasing the network depth. Algorithm 2 describes the steps of the Layer Pruner algorithm. ```
Input: \(\left\{w_{\ell}^{t}\right\}_{\ell=1\ldots\mathcal{L}^{t}}\): the learned layers' fuzzy relevance weights at iteration \(t\) Output: \(\mathcal{F}\Big{(};\mathcal{W}^{\mathcal{F}}\Big{)}\): the pruned network with the learned network depth \(\mathcal{L}^{t+1}\) \(max\_\ell\leftarrow\text{argmax}\big{(}\left\{w_{\ell}^{t}\right\}_{\ell=1 \ldots\mathcal{L}^{t}}\big{)}\)  Prune all layers after \(max\_\ell\)
```  
**Algorithm 2**Layer Pruning Algorithm  
Finally, Algorithm 3 summarizes the main steps of the proposed approaches, and Figure 2 shows the proposed approach flowchart. ```
Input: \(\left\{x_{j}\right\}_{j=1\ldots N}\): input instances \(\left\{y_{j}\right\}_{j=1\ldots N}\): the ground truth of the input instances \(\mathcal{F}\Big{(};\mathcal{W}^{\mathcal{F}}\Big{)}\): initialized deep feedforward neural network with initial depth \(\mathcal{L}\) Output: \(\mathcal{F}\Big{(};\mathcal{W}^{\mathcal{F}}\Big{)}\): trained deep feedforward neural network with learned depth \(\mathcal{L}^{Optimal}\) REPEAT  Select a batch \(B^{t}\) from \(\left\{x_{j}\right\}_{j=1\ldots N}\)  Train the network \(\mathcal{F}\Big{(}B^{t};\mathcal{W}^{\mathcal{F}}\Big{)}\)  Learn the layer fuzzy relevance weights using Algorithm 1  Prune layers using Algorithm 2 **UNTIL**\(\mathcal{F}\Big{(};\mathcal{W}^{\mathcal{F}}\Big{)}\) convergences
```  
**Algorithm 3**Depth-Adaptive Deep Neural Network  
```
Input: \(\left\{x_{j}\right\}_{j=1\ldots N}\): input instances \(\left\{y_{j}\right\}_{j=1\ldots N}\): the ground truth of the input instances \(\mathcal{F}\Big{(};\mathcal{W}^{\mathcal{F}}\Big{)}\): initialized deep feedforward neural network with initial depth \(\mathcal{L}\) Output: \(\mathcal{F}\Big{(};\mathcal{W}^{\mathcal{F}}\Big{)}\): trained deep feedforward neural network with learned depth \(\mathcal{L}^{Optimal}\) REPEAT  Select a batch \(B^{t}\) from \



## 4 Experimental Setting

### Datasets

The proposed method is assessed using the standard MNIST handwritten digits dataset [42] and CIFAR-10 tiny images dataset [43]. MNIST includes 60,000 training samples and 10,000 test samples distributed equally over 10 classes. Each sample consists of a 28 \(\times\) 28 grayscale image showing a single handwritten digit (0 to 9). Figure 3a shows sample images from the MNIST dataset. Alternatively, CIFAR-10 consists of 60,000 32 \(\times\) 32 color images distributed equally over 10 classes. The dataset is organized into 50,000 training images and 10,000 test images. Figure 3b shows sample images from the CIFAR-10 dataset representing the different classes. Figure 3: Sample images from benchmark datasets. (**a**) Sample images from MNIST dataset; (**b**) Sample images from CIFAR-10 dataset. Figure 2: Flowchart of the proposed approach. ### Experiment Descriptions

2.1 Experiment 1: Empirical Investigation of the Effect of the Plain Network Depth on the Model Performance

First, the performance of plain networks, including different numbers of hidden layers, is evaluated. Specifically, the number of layers is tuned from 1 to 50. Figure 4 illustrates a sample plain network that consists of an input layer and 50 convolutional layers followed by one dense layer. The details and settings of this plain network architecture are reported in Table 1. As can be seen, all layers have the same filter size, which is 32, and the kernel size equals (\(3\times 3\)).