# Dynamic Depth Learning in Stacked AutoEncoders
## 5 Experiments
### Experiment 1: Manual Depth Investigation  
The first experiment manually investigates the performance of the SAE with different depth values in order to show the effect of the SAE depth on the performance and thus prove the significance of learning the depth of SAEs. In other words, the performance of the SAE is evaluated with different depths. In fact, the depth of the model is tuned from one AE architecture to twenty AE architectures. To demonstrate the impact of the network depth on the classification performance, the experiment is conducted on the two-benchmark datasets MNIST [35] and CIFAR-10 [36]. After training the SAE, a SoftMax layer is appended on the encoder part as a classifier to perform the classification task. As the two benchmark datasets are balanced, the accuracy is considered. The latter with respect to different network depths is illustrated in Figure 5.  
As can be observed in Figure 5a, the best accuracy achieved is 87.20% with the one-AE model. Specifically, the classification accuracy on MNIST [35] showed that the one-AE architecture achieved a better classification accuracy compared with the 20-AE architecture. In fact, the obtained results showed that as the depth grows, the classification accuracy decreases. It is also observed that the performance of the model with depths from 1 to 4 attained better accuracy than models with deeper depths. In addition, the accuracy drastically decreases and plateaus for models with depth 13 to 20. This is a counterintuitive result showing that increasing the number of layers can decrease the accuracy.  
The performance on the CIFAR-10 [36] dataset is illustrated in Figure 5b. As can be seen, the model with two-AE architecture attained an accuracy of 25.84%. The achieved results confirm the findings on the MNIST dataset, which shows that as the depth increases, the classification accuracy does not necessarily increase. Moreover, the network with two-AE architecture performs better than other SAE depths.  
This steep fall in accuracy, shown in Figure 5, is related to the considered data. As can be seen, the curve displayed in Figure 5a is different from the one displayed in Figure 5b which depicts the accuracy achieved when manually tuning the depth on MNIST [35] dataset and CIFAR-10 [36] dataset, respectively. It is probably due to an overfitting problem when the depth is excessive. In fact, as a deep learning model, SAE is prone to overfitting. Specifically, as the number of layers increase, the number of network parameters that should be learned during the training phase increases. As such, it is essential to learn the optimal depth of an SAE that allows learning an abstract representation of the data while  
Figure 5: Accuracy achieved when manually tuning the depth on (**a**) MNIST dataset; and (**b**) CIFAR-10 dataset.  
\begin{table}
\begin{tabular}{c c c c c} \hline
**Parameter** & **MNIST** & **CIFAR** & **Parkinson** & **Breast Cancer** \\ \hline Optimizer & Adam & Adam & Adam & Adam \\ Loss Function & Binary Cross-Entropy & Binary Cross-Entropy & Mean Squared Error & Mean Squared Error \\ Learning rate & 0.0001 & 0.0001 & 0.000001 & 0.000001 \\ Batch size & 256 & 256 & 190 & 30 \\ \hline \end{tabular}
\end{table}
Table 3: Parameters setting.  
not being prone to overfitting. This is a motivation of the proposed approach which learns dynamically the depth while training the stacked AutoEncoder.