# Application of Deep Neural Networks in Recognition of Selected Types of Objects in Digital Images
## 2 Results of Research
### GoogLeNet  
The results shown in Figure 14 obtained using different activation functions in the GoogLeNet model show differences in model performance. The ReLU activation function achieved an accuracy of 65.71% and a loss of 1.35. Training with ReLU had a duration of 5257 s. Leaky ReLU, on the other hand, although it had a slightly longer training, improved  
Figure 12: Comparison of the results of the AlexNet model training process with different activation functions. (**a**) Results for ReLU. (**b**) Results for Leaky ReLU. (**c**) Results for ELU. Figure 13: Comparison of GPU and memory consumption during training of AlexNet model with different activation functions. (**a**) GPU and memory consumption for ReLU. (**b**) GPU and memory consumption for Leaky ReLU. (**c**) GPU and memory consumption for ELU. the model's accuracy to 67.41% and reduced the loss to 1.32. ELU, on the other hand, despite a longer training time (5638 s), brought a decrease in accuracy to 60.98% and a loss of 1.63. An analysis of GPU and memory consumption (Figure 15) shows that ReLU and Leaky ReLU showed similar GPU consumption, at 56.72% and 56.6%, respectively. In contrast, ELU, with lower GPU consumption (49%), proved to be less efficient in terms of model performance. The average memory consumption for the ReLU was 25.16%, while Leaky ReLU put more strain on memory, at 27.2%. ELU, on the other hand, despite lower memory consumption (23.12%), did not improve the overall performance of the model. _Learning rate optimization_  
In the process of training machine learning models, one of the key hyperparameters that affects the efficiency and speed of learning is the learning rate. The learning coefficient determines how many steps the model takes when updating the weights to minimize the cost function. Choosing the right learning coefficient is crucial: too large a coefficient can lead to an unstable learning process, while too small a coefficient can cause the model to learn very slowly and can get stuck in local minima. Optimizers are algorithms that are responsible for updating model weights based on the gradient of the cost function. Combined with an appropriately chosen learning rate, an optimizer affects the speed and stability of the learning process. Within the framework of this work, several different configurations of optimizers with different settings of the learning rate were subjected to study. The experiments conducted included the use of such optimizers as the following:  
Figure 14: Comparison of the results of the GoogLeNet model training process with different activation functions. **(a)** Results for ReLU. **(b)** Results for Leaky ReLU. **(c)** Results for ELU. * SGD with LR = 1 \(\times\) 10\({}^{-3}\): Stochastic Gradient Descent (SGD) optimizer with a fixed, constant learning rate of 0.001. SGD is one of the simplest optimizers that updates the model weights based on the average gradient calculated for each batch (batch). The fixed learning rate causes the model to learns at a steady pace throughout the training. * Adam with LR = 1 \(\times\) 10\({}^{-3}\): Adam optimizer, which is a more advanced version of the optimizer that combines the advantages of RMSprop and SGD with momentum. With a fixed learning rate of 0.001, Adam dynamically adjusts the weight update steps, which usually leads to faster and more stable convergence compared to classic SGD. * Adam with dynamically adjusted LR using LearningRateScheduler: This setup used a starting learning rate of 0.001, which was dynamically reduced by a factor of 0.1 after 7 epochs of training. This mechanism allows for faster learning at the beginning, and then stabilizes the process as the model approaches the minimum of the cost function. * Adam with LR reduced on Plateau: In this version, the learning factor was automatically reduced when it was noticed that the improvement in the value of the cost function stopped (i.e., when the value of val_loss stopped decreasing).