## 1 Introduction  
Convolutional neural networks (ConvNets or CNN) are biologically-inspired structures and represent a special type of neural network. CNN is most commonly used to process 2D signals for digital image processing, but it can also be used to process 1D signals. ConvNets are very successful in different visual tasks, such as classifying images, detecting objects in pictures [1], segmenting images [2], generating image descriptions [3], recognizing faces [4], pose estimation [5], and others. [PERSON] et al. proposed the first CNN in 1998 [6]. It was successfully applied to digit recognition. AlexNet [7] was the first architecture that achieved significant results, popularized CNN, and brought a revolution in computer vision. AlexNet has a very similar architecture to LeNet, but it is much deeper. Some of the other well-known architectures are VGG [8], and more modern networks are GoogleNet [9], ResNet [10], DenseNet [11], and SENet [12]. The fundamental building blocks of CNNs are the convolution layer, the pooling layer, and the fully-connected layer (FC layer). By stacking these layers properly in a deep network, it can successfully solve different visual tasks. The classic CNN takes an input image, then it performs the convolution operation by sliding filters through the input image and applies the activation function, which results in feature maps, and after that, the pooling layer is utilized to reduce the dimensionality. This process is repeated several times, and in the end, one or more fully-connected layers exist in the architecture. The last FC layer represents the classifier. More modern architectures employ other building blocks and different operations, such as the residual block [10], the inception module [9], and the squeeze-and-excitation block [12]. The performance of any CNN highly depends on the parameters such as the number of convolutional layers and filters, along with the filter size, batch size, etc. These parameters are known as the CNN hyperparameters [13]. The values of the hyperparameters determine the CNN's success rate for solving a particular problem, and a CNN architecture that uniformly generates satisfying results for all problem instances does not exist. Determining the right values of hyperparameters (hyperparameters' optimization) for a particular problem represents NP hard tasks and one of the greatest challenges from the CNN domain [14]. The CNN architecture design can be perceived as the model selection problem in machine learning [15]. According to the literature review, most researchers focused on the CNN architecture improvements concerning classification accuracy, and only some of them addressed the problem of the hyperparameters' optimization [15; 16; 17; 18; 19; 20]. The general hyperparameters in CNN that could be optimized are the following: the number of convolutional layers, the number of filters, the size of the filters, the stride, which is the number of steps that the filter takes when sliding through the input image, the type of padding, which can be either the same convolution or a valid convolution, the dropout probability, the batch size, and the number of training epochs. The choice of activation function, the regularization technique, significantly contributes to the network's performance. The most commonly used activation functions are sigmoid, hyperbolic tangent (TanH), and rectified linear unit (ReLU). The most popular hyperparameter tuning techniques are grid search and random search [21]. Grid search is a systematic process for hyperparameter optimization. It is a trial and error method, where every combination of hyperparameters is specified and evaluated on the model. This technique requires expertise from the domain. The grid search method can be effective when the number of hyperparameters is small. However, as the number of hyperparameters increases, the utilization of computational resources grows exponentially.