# Depth-Adaptive Deep Neural Network Based on Learning Layer Relevance Weights
## 3 Depth-Adaptive Deep Neural Network
### Layer Pruning
#### 4.2.2 Experiment 2: Performance Assessment of ANNA Approaches Applied to Plain Networks

The two ANNA approaches are applied to the 50 hidden layers' model architecture described in Table 1. In order to evaluate the performance of the two obtained models, the class-independent ANNA and the class-dependent ANNA are trained in order to learn the depth of the network, the relevance weight of each layer, and the model parameters. As result, both proposed models converged to one hidden layer-network for the two considered datasets, namely MNIST [42] and CIFAR-10 [43]. This is in accordance with the findings of the previous experiment. Thus, the proposed approaches are able to learn automatically the optimal number of hidden layers. This is achieved through the appropriate learning of the layers' relevance weights. #### 4.2.3 Experiment 3: Performance Assessment of ANNA Approaches Applied to ResNet Plain Models

In this experiment, the proposed ANNA approaches are evaluated using a plain baseline network inspired by VGG nets [44] and introduced in ResNet [36]. Specifically, for these models, layers having the same feature map size have the same number of filters. Alternatively, when the feature map size is halved due the increase in the convolution stride, the number of filters is doubled. In particular, two versions of these plain networks are considered. Namely, these are the 18 convolutional layer baseline, referred to as plain-18, and the 34 convolutional layer baseline, referred to as plain-34. Table 2 reports the detailed architectures of these two plain networks for the CIFAR-10 dataset [43]. Moreover, Figure 5 shows the block diagrams of the 18-layer and 34-layer plain baseline networks. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Layer Name** & **Output Size** & **50 Hidden Layers** \\ \hline Conv1 & \(32\times 32\) & \(3\times 3\), \(32\), stride 2 \\ \hline Conv2\_x & \(32\times 32\) & \([3\times 3\), \(32]\times 50\) \\ \hline Output & \(1\times 1\) & fully connected, softmax \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Detailed architecture of 50 convolutional layer baseline (Plain-50). Figure 4: CNN block diagram for 50 convolutional hidden layers. ResNet-18 [36] and ResNet-34 [36] are the residual versions of plain-18 and plain-34, respectively, where the shortcut connections are inserted. As such, the performance results of applying the two proposed ANNAs approaches to plain-18 and plain-34 models are compared to the performances of ResNet-18 and ResNet-34. This experiment is conducted on MNIST [42] and CIFAR-10 [43] datasets. The same experimental settings as the previous experiments are considered. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Layer Name** & **Output Size** & **18 Layers** & **34 Layers** \\ \hline Conv1 & \(19\times 19\) & \(7\times 7\), & 64, stride 2 \\ \hline \multirow{3}{*}{Conv2\_x} & \multirow{3}{*}{\(9\times 9\)} & \(3\times 3\),maxpool, stride 2 \\ \cline{3-4}  & & \(\begin{bmatrix}3\times 3\),64\\ 3\times 3\),64 \\ \end{tabular} \(\times 2\) & \(\begin{bmatrix}3\times 3\),64\\ 3\times 3\),64 \\ \end{tabular} \(\times 3\) \\ \hline Conv3\_x & \(5\times 5\) & \(\begin{bmatrix}3\times 3\), 128\\ 3\times 3\), 128 \\ \end{tabular} \(\times 2\) & \(\begin{bmatrix}3\times 3\), 128 \\ 3\times 3\), 128 \\ \end{tabular} \(\times 4\) \\ \hline Conv4\_x & \(3\times 3\) & \(\begin{bmatrix}3\times 3\), 256 \\ 3\times 3\), 256 \\ \end{tabular} \(\times 2\) & \(\begin{bmatrix}3\times 3\), 256 \\ 3\times 3\), 256 \\ \end{tabular} \(\times 6\) \\ \hline Conv5\_x & \(2\times 2\) & \(\begin{bmatrix}3\times 3\), 512 \\ 3\times 3\), 512 \\ \end{tabular} \(\times 2\) & \(\begin{bmatrix}3\times 3\), 512 \\ 3\times 3\), 512 \\ \end{tabular} \(\times 3\) \\ \hline Output & \(1\times 1\) & average pool, fully connected, softmax \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Detailed architecture of 18 convolutional layer baseline and 34 convolutional layer baseline models.