# Hyperparameter Black-Box Optimization to Improve the Automatic Classification of Support Tickets
## 3 Optimizing the Hyperparameters of the Classifiers
Due to the discrete nature of the majority of the choices, we use for this task a recently proposed algorithm [24] based on primitive directions and non-monotone line search which can deal with integer variables. Indeed, many hyperparameters assume integer values, or even categorical values which we encode with integers (e.g., the choice of the activation function of a neuron). Also, continuous hyperparameters are discretized, as it is often done, for example in a standard grid search. In particular, such an algorithm can guarantee theoretical convergence to a local optimum of Equation (1) by locally exploring the behavior of the objective function corresponding to variations in the optimization variables. This is obtained in practice by means of an iterative procedure evaluating the objective function over a sequence of points \((V^{\prime},V^{\prime\prime},\dots)\). Each new point is obtained from the previous one \(V^{i}\) by choosing a direction from a set of directions \(D(V^{i})\) which is specific to the current position, and by exploring the chosen direction by means of a non-monotone line search, to find a new point \(V^{i+1}\) that both guarantees a sufficiently large movement along the search direction and an improvement in the value of the objective function. To avoid getting stuck in points of the integer lattice that cannot be further improved, the set of possible directions is enriched by considering the so-called primitive directions as soon as the algorithm gets stuck. When moving over those points, the algorithm maintains the integrality of the variables and respects box constraints. Possible constraints delimitingthe feasible set are handled by using a penalty approach. The iterations are stopped either when the improvements become numerically negligible, or when an overall maximum number of hyperparameters evaluations is reached, or when a maximum number of evaluations without improvements is obtained. This algorithm is repeatedly restarted from several different starting points, to improve the quality of the final solution (multi-start technique). To evaluate the advantages of the above optimization approach, we compare it to the selection of the values of the hyperparameters by using a standard technique called _grid search_, which is very often used in the literature. A grid search is the simple evaluation of all the combinations of the possible hyperparameter values. This corresponds to a complete enumeration approach in the solution of problem 1. If not interrupted, it clearly guarantees completeness, hence in theory it always reaches the optimal solution, however, this may require in practice very long times, often excessive. For each tested classifier, we list below all the values of the hyperparameters that were considered in our experiments. Due to the time required by each hyperparameter evaluation (full training and testing with 5-fold cross-validation), we could experiment with only the sets of values for the hyperparameter that were deemed sufficiently promising. However, in principle, our technique can handle any set of hyperparameter values, provided that the required computations can be carried out in practice. Furthermore, since the number of points tested in a grid search tends to be exponential (all the possible combinations), for mere computational reasons we are forced to reduce in the grid search the number of values tested for each hyperparameter with respect to those considered in the optimization approach. In particular, we consider two variants of the grid search, which we call \"Grid Search\" and \"Extended Grid Search\". The first is intended to run in a reasonable time, thus the size of the search space, fixed in advance, must be quite smaller than that of the Black Box procedure. The second is intended only as a validation of our Black Box procedure, and not as a practically useful alternative.