## 4 Scenarios
### Extract Features with CNNs
#### 4.2.1 Extract Features without Retraining  
Depending on the depth of the layer, the CNNs extract features of different sizes and spatial resolution. We extract features from four different depths of the CNNs, specifically from 25%, 50%, 75%, and 100% of their depth, to identify whether the features extracted from shallower levels are more appropriate than from the output layer.  
Figure 4 depicts the results of the classification accuracy per set (blue bar for KDEF, red bar for JAFFE, and green bar for RaFD), per network, and per depth. Extracting features from the last and deepest layer of networks (i.e., from 100% of the depth) leads to the worst results in terms of classification accuracy, with success levels being much lower than those of the handcrafted methods. On the contrary, at 25%, 50%, and 75% of the depth of the networks, the classification accuracy reaches or exceeds the classification accuracy of the handcrafted methods depending on the network used.  
At 25% of the depth, the largest in-depth network, ResNet101, performs better than the other networks, giving better results in all three databases. At 50% of the depth, the ResNet family technique performs better, and specifically, ResNet50 gives similar results to those of ResNet101 at 25%, which was to be expected since the difference in the percentage of depth combined with the total depth of the networks gives features of the same size. At 75% of the depth, the results are inferior to the previous ones except for the JAFFE dataset, for which we have the highest performance with ResNet18. Table 5 summarizes the highest classification accuracy per method and the corresponding computation time taken by the  
Figure 3: Highest classification accuracy achieved for each database with handcrafted methods.  
algorithms to extract the features and classify the test set. For the cases where we had the same classification percentage with different networks and depths, the selection was made based on the shortest time for exporting results.  
Up to this point, extraction of features from ResNet50 without retraining in the new databases from 50% of its depth performs better in comparison with the handcrafted methods in terms of execution time and classification accuracy for the KDEF and RaFD  
Figure 4: Classification accuracy as a function of CNN and depth for each database, without retraining with the SVM classifier. Dashed lines depict the highest accuracy achieved with the previous (handcrafted) methods.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & \multicolumn{2}{c}{**Handcrafted Methods**} & \multicolumn{2}{c}{**CNNs**} \\
**Database** & **CA (\%)** & **Technique** & **Time (s)** & **CA (\%)** & **CNN and Depth** & **Time (s)** \\ \hline KDEF & 78.47 & (LPB \(8\times 8\)) & 1623 & 81.21 & (ResNet50, 50\% depth) & 1214 \\ JAFFE & 85.71 & (HOG 16 \(\times\) 16) & 5 & 92.86 & (ResNet18, 75\% depth) & 55 \\ RaFD & 94.40 & (LPB 16 \(\times\) 16) & 3746 & 95.71 & (ResNet50, 50\% depth) & 2988 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Highest classification accuracy and the corresponding total time per method and database.  
databases. For the JAFFE database, classification accuracy is also significantly improved, with a slight increase in the execution time required by the ResNet18 at 75% of its depth.