## 8 Implementation and Experiments
### Deep Learning Experiments
We experimented with networks of varying depth (16 layers at most) and various structural ideas: dense layers alone, single convolutional layers in front of pooling layers, blocks of stacked convolutional layers (VGG-like structures), varying filter sizes and counts of output channels, dropout rates, etc. We must emphasize that the main goal of these experiments was _not_ to obtain excellent classifiers for the mentioned data sets. Instead, the goal was to verify the _consistency_ of results from Keras and the \"home-made\" implementation, when the same architectures were applied. We remark that both implementations used a uniform He initialization of weights for layers with ReLUs, a uniform Glorot initialization otherwise, and the Adam optimizer (\(\eta=10^{-3}\), \(\beta_{1}=0.9\), \(\beta_{2}=0.999\), and \(\epsilon=10^{-7}\)). Finally, we need to remark that, owing to the imposed randomization seeds the reported accuracies and loss values can be _reproduced exactly_ for a fixed GPU device. We imposed seed=0 as the default value across several modules--in particular, we executed random.seed(seed), tf.random.set_seed(seed), and np.random.seed(seed). Also, please see the set_tf_global_determinism(seed) function, where we forced the deterministic behavior of TensorFlow and CUDA; additional data set- or repetition-related seeds can be changed by the user via the data_description dictionary and its entry named \"seed\", placed in the main program function. One should realize that results do differ between devices due to the order and properties of computations carried out on the GPU (e.g., the number of streaming multi-processors and CUDA cores, scheduling scheme, etc.). Figure 27: Sample images from data sets applied in experiments. _Appl. Sci._**2024**, 14, 9972  
\begin{table}
\begin{tabular}{c c c c c c c c c c c c c} \hline \hline \multicolumn{11}{c}{**Data Set: Olivetti**} \\ \hline \multicolumn{11}{c}{**No. **} & \multicolumn{1}{c}{**Structure String**} \\ \hline \multicolumn{11}{c}{**SO**} & \multicolumn{1}{c}{**Expt. ID**} & \multicolumn{1}{c}{**Fit**} & \multicolumn{1}{c}{**Fit**} & \multicolumn{1}{c}{**Fit**} & \multicolumn{1}{c}{**Test**} & \multicolumn{1}{c}{**Time**} & \multicolumn{1}{c}{**Test**} & \multicolumn{1}{c}{**Test**} & \multicolumn{1}{c}{**Test**} & \multicolumn{1}{c}{**Test**} \\ \multicolumn{11}{c}{**No. **} & \multicolumn{1}{c}{**Structure String**} & \multicolumn{1}{c}{**(hash)**} & \multicolumn{1}{c}{**Params**} & \multicolumn{1}{c}{**[s]**} & \multicolumn{1}{c}{**Acc. **} & \multicolumn{1}{c}{**Loss**} & \multicolumn{1}{c}{**Is]**} & \multicolumn{1}{c}{**Acc. **} & \multicolumn{1}{c}{**Loss**} & \multicolumn{1}{c}{**Is]**} & \multicolumn{1}{c}{**Acc. **} & \multicolumn{1}{c}{**Loss**} \\ \hline \multicolumn{11}{c}{The overall results confirm several well-known or intuitive facts. CIFAR-10, the hardest among the data sets, allowed only a few structures to surpass 80% accuracy, whereas it was fairly easy to beat 99.2% for MNIST. For these two sets, it is also easy to see that dense layers alone are not sufficient for good accuracy--convolutional layers are needed. Furthermore, blocks of _stacked_ convolutional layers (VGG-like) led to the best results. Due to the small  
size of the Olivetti data, the above observations were not evident on that set. The results oscillated around 94% \(\pm\) 2% with little dependence on the structure. In every row of Tables 1-3, the blue color is used to mark the higher accuracy and the green to mark the smaller loss. The last two columns report _relative differences_ in the results between implementations, i.e., higher/lower accuracy \(-1\), and \(1-\) smaller/larger loss (as a percentage). As regards accuracy, the results are highly _consistent_--for a large majority of cases, the relative difference is smaller than 1%, with few outliers (the worst: 3.54%). Naturally, categorical cross-entropy loss is a more sensitive measure than accuracy.