# Dynamic Depth Learning in Stacked AutoEncoders
## 5 Experiments
### Experiment 3: Performance Assessment of the Proposed Approach DDSAE on Real Datasets  
To assess the performance of the proposed approach DDSAE in dynamically learning the depth of SAE, two real datasets, Parkinson's [37,38] and Breast Cancer [39,40], are considered. As in experiment 2, the initial number of AEs was set to 20. DDSAE converges to one AE layer for both datasets. These results are in accordance with the findings of experiment 2. Figure 10 shows the depth learned dynamically (the retained number of layers) while training each batch on the Parkinson's [37,38] and Breast Cancer [39,40] datasets. As can be seen, the depth decreased from 20 AEs (after training the first batch) to one AE (after training the second batch) for both datasets. During the training phase, the layer relevance weights are estimated in an unsupervised way at the end of each batch using the derived equation defined in Equation (3) subject to Equations (4) and (5). Giving insight into the importance of each layer, they are employed to dynamically prune irrelevant layers exhibiting low layer relevance weight. Figures 11 and 12 depict the layer relevance weights learned by DDSAE on the Parkinson's [37,38] and Breast Cancer [39,40] datasets in the first and last batch, respectively. As can be seen, after training the model using one batch, the maximum layer relevance weight is assigned to the first layer of the encoder. Hence, all layers exhibiting a relevance weight smaller than 75% of the maximum weight are pruned. In terms of learnable parameters, the number of learned parameters on Parkinson's [37,38] has dropped from 41,547,753 to 3,509,753 after the convergence of the model to one AE network. In addition, the learned parameters on Breast Cancer [39,40] decreased after convergence from 40,059,009 to 2,021,009 parameters. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Dataset** & \multicolumn{2}{c}{**MNIST**} & \multicolumn{3}{c}{**CIFAR-10**} \\ \hline Classifier & SoftMax & SVM & KNN (k = 5) & SoftMax & SVM & KNN (k = 5) \\ Accuracy & 97\% & 95.78\% & 95.93\% & 38\% & 33.39\% & 28.45\% \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Accuracies achieved on MNIST and CIFAR-10 datasets. Figure 10: Number of retained layers with respect to each iteration on (**a**) Parkinson’s dataset; and (**b**) Breast Cancer dataset. Moreover, the performance of DDSAE is assessed by appending different classifiers on top of the encoder part after training. Since the datasets are unbalanced, precision, recall, and F1-measure are reported in addition to the classification accuracy. Table 5 reports the performance attained on the Parkinson's [37; 38] and Breast Cancer [39; 40] datasets using different classifiers. As can be observed, the performance on Parkinson's with the KNN and SVM classifiers achieved similar results; however, the Soft-Max classifier achieved lower results. Similar results were obtained on the Breast Cancer dataset. Spherically, DDSAE reached an F1-score of 77% on the Parkinson's [37; 38] dataset and an F1-score of 89% on the Breast Cancer [39; 40] dataset when using the KNN classifier. Therefore, in addition to learning the optimal depth of the SAE, DDSAE is able to yield good classification results. Figure 11: Layer relevance weights learned by DDSAE on Parkinson’s dataset after training the model using (**a**) first batch; (**b**) last batch. Figure 12: Layer relevance weights learned by DDSAE on Breast Cancer dataset after training the model using (**a**) first batch; (**b**) last batch.