# Multi-Input Convolutional Neural Networks for Automatic Pollen Classification
## 2 Materials and Methods
### Network Architecture
#### 2.2.4 Hyper-Parameter Tuning  
A standard procedure to increase the performance of any machine learning algorithm is selecting good hyperparameters for the specific task. The number of hyperparameters that are available for tuning depends on the complexity of the proposed approach, with convolutional neural networks having a very large number of possible values to choose from. Finding a good hyperparameter tuple in a very large search space is difficult and computationally expensive; therefore, the task was split into multiple smaller problems. Thanks to the modularity of the network and the possibility of training each component independently, the hyperparameter search was conducted separately for each feature extractor. This reduced the dimensionality of the search space by orders of magnitude and increased the chances that good hyperparameters could be found. In Table 3, the values selected for the hyperparameter search are presented. For most parameters, the values were selected considering a linear range, except for the learning rate, which used a logarithmic scale to better probe the search space. The last column of the table presents the setup used in [9]. When searching for a good combination of hyperparameters, a grid search [19] is guaranteed to eventually give the best results, if a small-enough step is selected. The only problem with this approach is that it takes huge amounts of time to train all the different variations of the models. This approach is only applicable if the number of hyperparameters is really low. Another approach is using a random search. This method was shown to be good enough with a much smaller impact on training time [20]. Hyperparameter values are chosen at random from the available ranges and a model is trained. After training, the model is evaluated and the parameter and score are saved. Even better than a random search is a guided approach, such as Bayesian optimization [21]. This approach tries to optimize over the search space by first constructing a posterior distribution of functions that best describes the function to be optimized. Then, as the number of observations grows, the posterior distribution improves and the algorithm becomes more certain of which regions in the parameter space are worth exploring and which are not. The Bayesian optimization was performed using a Python package developed by [22]. This package was used to first randomly sample the search space to initialize the algorithm. After this first step, the package recommended the hyperparameters based on metrics that balanced exploration versus exploitation of the known features of the search space. Hyperparameter tuning was performed independently for each feature extractor, with each tuning run consisting of 10 initial random selections followed by 200 guided selections. As can be seen in Table 4, the parameters selected after the search are very different depending on the feature in question. For the scattering image feature extractor, a higher learning rate was selected than for the other two. The selected batch size was more than double for the spectrum feature than for the lifetime one. The number of neurons in the dense part of the feature extractors appears to be correlated with the drop-out rate. The filter kernel size chosen was \(5\times 5\) for scattering and spectrum and 3 for the lifetime signal. Looking at the number of filters per layer, we can see that, even though the possible range was 4-128, only the lifetime feature extractor went in the direction of many filters, with the other two using less filters than the initial architecture.