# Hyperparameter Black-Box Optimization to Improve the Automatic Classification of Support Tickets
## 4 Experimental Results
The extraction has been randomly performed 5 times, and all performance results are averaged on the 5 trials. The resulting 7-class problem is not trivial. Just as an example, preliminary tests with some \"default\" hyperparameter values obtain less than 50% in accuracy. As reported in Tables 1 and 2, by considering all the values of the hyperparameters described in the previous Section, in the case of CNN we obtain 5760 possible hyperparameter configurations for the black-box optimization approach, 240 hyperparameter configurations for the grid search approach, and 576 hyperparameter configurations for the extended grid search. Hence, the search space of the extended grid search is one-tenth of the black box, however, it was impossible to extend it further, since this already requires 250 h of computation (more than 10 days). Indeed, if the grid search was extended to the full search space of the black box, the required time could be roughly estimated as (375,000/240) \(*\) 5760 = 9,000,000 s, which is about 104 days, in the case of CNN. In the case of SVMs, we obtain 49,780 possible parameter configurations for the black-box optimization approach and for the extended grid search, and 1280 parameter \begin{table}
\begin{tabular}{l r r r} \hline  & **Black Box** & **Grid Search** & **Ext. Grid Search** \\ \hline Hyperparameter configurations & 5760 & 240 & 576 \\ Evaluated points & 212 & 240 & 576 \\ Enumeration percentage & 2.76\% & 100\% & 100 \% \\ Time in sec. & 318,500 & 375,000 & 901,000 (\textgreater{}10 days) \\ Solution accuracy & 89.72\% & 86.15\% & 89.75\% \\ \hline \end{tabular}
\end{table} 
Table 1: Comparison of black-box optimization and 2 variants of grid search for CNN. \begin{table}
\begin{tabular}{l r r r} \hline  & **Black Box** & **Grid Search** & **Ext. Grid Search** \\ \hline Hyperparameter configurations & 49,780 & 1280 & 49,780 \\ Evaluated points & 291 & 1280 & 49,780 \\ Enumeration percentage & 0.58\% & 100\% & 100\% \\ Time in sec. & 3490 & 15,360 & 597,360 (\textasci{}7 days) \\ Solution accuracy & 74.53\% & 71.87\% & 74.53\% \\ \hline \end{tabular}
\end{table} 
Table 2: Comparison of black-box optimization and 2 variants of grid search for SVM. configurations for the grid search. In this case, the extended grid search can cover the same search space of the optimization approach. In conclusion, the size of the search space of the optimization approach may be quite larger than that of the grid search. However, only a small fraction of the points of this search space is actually evaluated by the optimization approach (see Evaluated points in Tables 1 and 2), so this last approach can find a solution in less time. The tables also report the percentage of points evaluated by each approach (Enumeration percentage). In the case of our 7-class problem, we select, as an appropriate performance metric, the accuracy, defined as the overall percentage of correct class predictions over the test set. Hence, in our experiments, we pursue maximum accuracy both during the black-box optimization and during the grid search. Tables 3 and 4 report respectively the best hyperparameter configurations obtained in the case of CNN and SVM. Clearly, the extended grid search guarantees us to find the best accuracy within its search space, and it was intended to validate the result of the optimization approach. It turns out that the solution obtained by the black-box optimization approach is, for both classifiers, fully comparable with that of the extended grid search. In particular, it is slightly different but with almost the same value in the case of CNN, and it is exactly the same in the case of SVM. However, the time required by the optimization procedure is only a small fraction of that of the extended grid search. On the other hand, the comparison between the optimization approach and the version of grid search requiring a more reasonable time gives slightly different results. Therefore, the advantages of the proposed optimization approach with respect to the grid search can be described as follows. 1.