# Performance of Fine-Tuning Convolutional Neural Networks for HEp-2 Image Classification
## 3 Materials and Methods
### Hyperparameter Optimization  
The design of a generic CNN provides, in addition to establishing the architecture and the topology, the optimization of parameters and connection weights through optimization procedures in the training phase. It refers to the numerous values to be optimized as hyperparameters optimization.  
Figure 3: Schematic of the AlexNet architecture. The output has been replaced considering the two classes on which to discriminate.  
The pre-trained CNNs have a consolidated and optimized architecture for the database for which they were originally trained.  
The process of optimizing the parameters during the training phase is certainly not trivial. Among the most recognized methods, there is the Stochastic Gradient Descent with Momentum (SGDM) that minimizes the loss function at each iteration considering the gradient of the loss function on the entire training dataset. The momentum term reduces the oscillations of the SGDM algorithm along the path of steepest descent towards the optimum. The momentum is responsible for reducing some noise and oscillations in the high curvature regions of the loss function generated by the SGD. A variant of the SGD uses training subsets called mini-batches, in this case a different mini-batch is used at each iteration. Simply put, the mini-batch specifies how many images to use in each iteration. The full pass of the training algorithm over the entire training set using mini-batches is one epoch. SGDM with mini-batch was used in this work.  
Another fundamental parameter of the training process is the learning rate that allows to set the learning speed of the training process with the level of improvement of the network's weights. Conceptually, a high learning rate increases the speed of training execution by sacrificing the performance of the trained network, while a low learning rate will increase the training times by optimizing the network's weights that will be more performing. This parameter defines the level of adjustments of weight connections and network topology applied at each training cycle. A small learning rate permits a surgical fine-tune of the model to the training data, at the cost of a greater number of training cycles and longer processing times. A high learning rate permits the model to learn more quickly, but may sacrifice its accuracy caused by the lack of precision over the adjustments. This parameter is generally set to 0.01, but in some cases, it is interesting to be fine-tuned, especially when it is necessary to improve the runtime when using SGD. Table 2 shows the search space analyzed for the optimization of the parameters.  
The extraction of features from a pre-trained CNN is simple to implement and computationally fast as it exploits the implicit power of representing CNN characteristics. In this work, the feature vector extracted from the generic CNN is used to train a SVM classifier. As the size of the feature vector is large, we have chosen to train a linear SVM that has one hyperparameter: the penalty parameter \"C\" of the error term. The search for linear kernel parameter \"C\" is carried out using MATLAB \"logspace\" function in the range [\(10^{-6}\), \(10^{2.5}\)]; twenty equidistant values on a logarithmic scale were analyzed.