# Hyperparameter Black-Box Optimization to Improve the Automatic Classification of Support Tickets
## 3 Optimizing the Hyperparameters of the Classifiers
### Convolutional Neural Networks  
The deep strategy used for our classification is a Convolutional Neural Network (CNN). In general, an artificial Neural Network is a collection of connected nodes called artificial neurons, which resemble the neurons in a biological brain. These artificial neurons are typically organized in layers, according to different architectures, and the resulting network can perform classification tasks by training the neuron input weights to minimize misclassification errors. In particular, CNN includes internal layers that perform convolutions. Such convolutional filters are sometimes called kernels. These networks are often used in image recognition or classification because they behave similarly to neurons in the visual cortex of animals. However, their classification capability can also be successfully applied to texts (see, e.g., [26]), since this type of architecture can extract \"deep\" features of the data, i.e., distinctive local motifs, regardless of their position in the data record. In more detail, we build a CNN with three layers of type 2D convolutional, each followed by a 2D Max Pooling layer, and one Dense layer in output. We choose this architecture as a good compromise between speed and performance, since the solution of problems 1 requires us to train and execute the network a large number of times with many different sets of hyperparameter values. The hyperparameter tested for this CNN are:  
* Embedding_dim: width of the kernel matrix for the 2D convolution window. The considered values are (200; 300; 400; 500), both for the optimization approach and for the grid search approach. For the extended grid search, to keep the computational time within 250 h (\(>\)10 days) and explore in detail the other hyperparameters, we are forced to consider only (200; 300) which are however the most promising values. * Filter_sizes: height of the 2D convolution window; in other words, the number of words we want our convolutional filters to cover. We specify 3 different values for the three layers. The values tested with the optimization approach are ([3,4,5]; [5,4,3]; [3,5,7]; [7,5,3]). For the grid search approach, for the computational reasons explained above, we use only ([3,4,5]). For the extended grid search we use ([3,4,5]; [5,4,3]). * Num_filters: the number of output filters in the convolution. We consider for all the approaches only the value 512, since preliminary experiments showed not much sensitivity to this parameter. * Optimizer: the optimization technique used in the gradient descent when training the network. For all the approaches we consider (adam; adamax; RMSprop; sgd). * Loss_function: the function used to calculate the error when training the network. For our multi-class problem, we selected only categorical cross-entropy as the loss function in all the approaches, since it was deemed the most suitable. * Activation_Conv: activation function for the convolutional layers. The activation functions of a neuron determine whether it should be activated (\"fired\") or not, based on the inputs received. Many activation functions exist, see also [27]. They must also be computationally efficient because they are calculated across many neurons for each data sample. For the optimization approach, we use (linear; relu; elu; selu; softsign; softplus; sigmoid; hard_sigmoid; exponential; tanh). For the grid search approach, we slightly limit the choice to the most promising and use (relu; softsign; sigmoid; exponential; tanh). For the extended grid search, we use (relu; elu; softsign; sigmoid; exponential; tanh). * Activation_Dense: activation function for the final dense layer. For the optimization approach, we use (relu; softmax; sigmoid). For the grid search approach, for the computational reasons explained above, we only use (softmax). For the extended grid search, we use (softmax; sigmoid). * Epochs: an epoch is one complete pass through the training data.