## 8 Implementation and Experiments
### Deep Learning Experiments
See, e.g., MNIST classifiers, where high accuracies of 99.4% or 99.5% are accompanied by losses ranging from about 0.016 to 0.032. Thus, relative loss differences are naturally of a larger magnitude. For the same architectures, the results of hmdl and Keras are highly consistent in terms of accuracy. But one might be also interested in simply counting the times that hmdl or Keras actually returned a better model, neglecting the margin and relative difference in accuracy (typically smaller than 1%). For the Olivetti data, hmdl did so in 7 out of the total of 22 executions (hence, the Keras models were better in 15 cases). For MNIST, hmdl produced better models in 6/10 cases, whereas for CIFAR-10, the hardest data set, it did so in 15/24 cases. As regards some particular results, the following observations can be made. For the MNIST data set, the structures of neural networks from experiments 7-10 contained the aforementioned VGG blocks, i.e., stacks of convolutional layers (denoted by \(\mathbb{C}(\cdots)\times 2\) strings), followed by the maximum pooling layers. Those structures translated into higher accuracies (typically surpassing 99.40%) than the structures from experiments 1-6, without the stacked convolutions. A similar observation is true for experiments on the CIFAR-10 data set, where the gap between such groups of structures is even more noticeable. In experiments 8-12, all networks involving the stacked convolutions surpassed an accuracy of 76% (reaching \(83\%\pm 1\%\) in a few cases), whereas all networks from experiments 1-7, without stacked convolutions, had trouble reaching a 70% accuracy level. Some of the experiments were planned to verify whether dropout layers distributed along the structure translate into better results when working with constant rates (e.g., equal to 0.125) or rates increasing towards the \"right-end\" of a network (e.g., 0.125, 0.25, and 0.5). In the case of the MNIST data, no clear difference in this respect was observed. Interestingly, for the CIFAR-10 data, models with increasing dropout rates (experiment no. 12) performed significantly better than their counterparts with constant rates (experiment no. 11). Differences of approximately 2% in accuracy were noticed. This observation was true for models that came from both frameworks (Keras and hmdl) and from both GPU devices (Quadro M4000M and A100). Finally, as regards comparisons of filter sizes for convolutions (\(7\times 7\), \(5\times 5\), and \(3\times 3\)) spread across various experiments, we did not observe any clear influence of filter size on accuracy. On the other hand, in the large majority of cases, larger filter sizes translated naturally into a longer time of model fitting. In this context, it is worth mentioning some of the recent results, from 2021 to 2024, achieved for the MNIST and CIFAR-10 benchmarks using the newest non-standard deep learning ideas. As regards MNIST, the results based on the so-called _capsule networks_ stand out. For example, [PERSON] [42] reports an accuracy of 99.84% using a capsule network with self-attention routing (Efficient-CapsNet) involving only 161 k trainable parameters. On the other hand, [PERSON] et al. [43] apply a similar approach, but they claim that no routing between capsules is needed. Their model achieves a 99.87% accuracy but employs significantly more trainable parameters--1.5 M. As regards CIFAR-10, recent results differ quite significantly, depending on the new ideas researchers experiment with. For example, [PERSON] et al. [44] propose an interesting idea of \"scalable hierarchical aware CNN\"(SHA-CNN). They carry out tests on MNIST, CIFAR-10, and CIFAR-100, but the accuracy achieved for CIFAR-10--83.35%--is not very high. In fact, some of our CNNs obtained using Keras or hmdl surpass that result. On the other hand, there exist recent articles that report outstanding accuracies for CIFAR-10. For example, [PERSON] [45] applies the so-calledvision transformers_ and obtains an accuracy of 99.3%.