## 8 Implementation and Experiments
### Deep Learning Experiments  
All experiments reported in this section can be reproduced by the reader using the script experimenter.py. It allows training (optional) and testing to be run for one of three well-known data sets, Olivetti ([[https://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html](https://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html)]([https://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html](https://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html))), MNIST ([[https://yann.lecun.com/exdb/mnist/](https://yann.lecun.com/exdb/mnist/)]([https://yann.lecun.com/exdb/mnist/](https://yann.lecun.com/exdb/mnist/))), or CIFAR-10 ([[https://www.cs.toronto.edu/~kriz/cifar.html](https://www.cs.toronto.edu/~kriz/cifar.html)]([https://www.cs.toronto.edu/~kriz/cifar.html](https://www.cs.toronto.edu/~kriz/cifar.html))) (accessed on 22 October 2024) [69, 70, 71] (see Figure 27), using two DL frameworks, hmdl and Keras. For every conducted experiment, we have generated a distinct 10-digit _identifier_, hashed based on given network structure and an additional description (data name, initial randomization seed, and number of repetitions). The log files and classifiers obtained can be found in the repository (folders experiments/, hmdl_clfs/, keras_clfs/) with their names prefixed using the appropriate identifier. Detailed instructions on how to run a new experiment or reproduce an existing one (when knowing its identifier) were moved to Section 8.8. Two hardware environments were applied to conduct the experiments--the first one was a personal computer equipped with NVIDIA Quadro M4000M as the GPU device (details on page 42), and the second was a high-performance server equipped with the NVIDIA A100 GPU for virtualized computations (details provided below). We explain that the complete set of all experiments was carried out independently in each of the two environments. _\"GRID A100-7-40C MIG 7g.40 gb\" environment--_hardware: Ubuntu Server 20.4.03, 4 CPUs: AMD EPYC 7H12 64-Core (2.6 GHz), 128 GB RAM, and NVIDIA GRID A100-7-40C vGPU,  
Figure 26: Implementation of Adam (coupled with regularization) in the hmdl project. manufactured by: Supermicro (Super Micro Computer, Inc.), San Jose, CA 95131, USA; software: nvcc 11.4 (V11.4.48), Python 3.8.10, numpy 1.23.5, numba 0.56.4, cv2 4.7.0, keras 2.12.0, and tensorflow 2.12.0; OS: Ubuntu 20.04.3 LTS. Tables 1-3 gather all the results. In strings representing the structures of tested neural networks, the following notation was applied. The abbreviations \"c\", \"w\", \"r\", \"p\", and \"DR\" stand for different layer types: convolutional, max pooling, flattening, dense, and dropout, respectively. The parameters of a layer are written in parentheses. For example, C(3, 32, r) stands for a convolutional layer using a \(3\times 3\) filter and producing 32 output channels (kernels) activated via the ReLU function, denoted as \"r\". DR(0.125) stands for a dropout layer using the rate of 0.125. M(2) represents a max pooling layer that applies a \(2\times 2\) regions to produce a down-sampled output tensor. D(10, sm) represents a dense layer with 10 neurons activated via the softmax function. The symbol \"\(\times 2\)\" denotes a stack of the same two layers. Due to the small size of the Olivetti data (400 examples in total), its experimental executions were repeated 10 times (with successive randomization seeds starting from zero, and the data were split randomly as 70:30 into training and test parts), and the results were averaged. MNIST and CIFAR-10 are moderately large sets (60 k and 50 k training examples, respectively, and 10 k testing examples for both); therefore, their experiments were conducted just once. For all data sets, we imposed 100 epochs of training but different counts of batches per epoch (n_batches parameter in hmdl library): 10 batches for Olivetti, 20 for MNIST, and 50 for CIFAR-10. These counts were suitably translated into the batch_size parameter in Keras.