# Hyperparameter Black-Box Optimization to Improve the Automatic Classification of Support Tickets
## 3 Optimizing the Hyperparameters of the Classifiers  
After obtaining the training set of labeled tickets, we set up a classification phase, which can predict the class of unlabeled tickets by learning the classification criteria from the above training set. We perform experiments with both _deep_ and _traditional_ learning strategies. In particular, we use Convolutional Neural Networks (CNN) [21] and Support Vector Machines (SVM) [22] classifiers. In general, to use each of these classifiers, a tuple of values \(V=(v_{1},...,v_{n})\) must be assigned to a set \(H\) of _hyperparameters_. Note that, for simplicity, we call here \"hyperparameters\" all the values needed by the classifier which are not learned by data, not distinguishing between the categories of those values. On the contrary, the values learned by data (e.g., neuron weights in a neural network) are generally called only \"parameters\"  
Figure 1: Overall scheme of the proposed approach, including both the text mining phase and the classification phase with hyperparameters optimization. and will not be set in our hyperparameters optimization but will be determined during the training of the classifier, which constitutes an \"inner loop\" of our procedure (see also Figure 1). Generally speaking, the majority of the hyperparameters are bounded to take integer values (e.g., the number of neurons in a neural network), even if some can vary with continuity within a given interval (e.g., the kernel coefficient \(\gamma\) in a support vector machine), and some are categorical (e.g., the choice of the activation function of a neuron). Thus, each value \(v_{i}\) must belong to its feasible domain \(D_{i}\), and consequently, we can define the set \(\mathcal{H}\) of all the feasible tuples of hyperparameter values. The choice of a tuple of values, that is, the choice of a point in \(\mathcal{H}\), determines the behavior of the classifier, and so it may dramatically affect its prediction performance \(P\). As a matter of fact, small variations in hyperparameter values may sometimes determine a huge variation in \(P\). Therefore, the values assigned to the hyperparameters must be selected very carefully, and they cannot simply be \"default\" values or similar. We propose to view the choice of these values as a higher-level optimization problem, where the hyperparameters \(H\) are the decision variables and the objective is the performance \(P\) of the classifier, evaluated by choosing a performance measure appropriate to the specific classification case. \[ \begin{array}{c}\max\ P(H)\\ H\in\mathcal{H}\end{array}  \tag{1}\]  
In our case with 7 classes, the accuracy (defined as the overall percentage of correct class predictions over the test set) was deemed to be the appropriate measure, even if in other cases one may prefer precision, sensitivity, F-1 score, etc. However, the above problem has a main difficulty. Even though \(P\) clearly depends on \(H\), this dependency cannot be expressed in analytical form. This is not only due to the complexity of such a relationship, but also because the performance \(P\) depends not solely on the rules governing the classifier, but also on the data handled by it. Hence, an explicit optimization model cannot be written for Problem 1. To solve despite this issue, we propose to adopt a black-box optimization approach and use a derivative-free algorithm. Similar algorithms do not need the explicit optimization model, in particular, the analytic expression of the objective function; they only need to numerically evaluate the objective function over a number of points (i.e., tuples of values of the hyperparameters). In practice, to evaluate a set of hyperparameter values \(V^{\prime}\), we run the classifier on the dataset using values \(V^{\prime}\), doing both training and testing (using 5-fold cross-validation in our experiments), and so we obtain the classification performance \(P^{\prime}\) corresponding to \(V^{\prime}\).