# Hyperparameter Black-Box Optimization to Improve the Automatic Classification of Support Tickets
## 1 Introduction
Many words may be in common between messages dealing with completely different subjects, and on the other hand, messages dealing with the same subject may use completely different words. Hence, grasping the semantics may be tricky. Several approaches to multi-class classification exist, based on different models and paradigms. However, to obtain a practically useful subdivision of the tickets, the hyperparameters of the selected classification algorithm should be carefully tuned. Indeed, as recently highlighted by several researchers, the selection of these hyperparameters is often the weak link in many machine learning applications [11; 12], because the difference in the classification performance between a \"good\" and a \"bad\" hyperparameter choice can be dramatic, especially for hard problems. For this reason, in the recent subfield of Auto Machine Learning, the values of those hyperparameters are searched by automatic procedures. This constitutes an important meta-learning step. For example, [13] presents hyperopt, a reusable software engine for hyperparameter optimization which could outperform domain experts in the tuning of Deep Belief Networks (DBNs). [PERSON] [14] observes that hyperparameter tuning may be very computationally demanding, and proposes a recommender system based on meta-learning to exactly identify when to use default values and when to tune hyperparameters, focusing the analysis on Support Vector Machines and extending it to decision trees. In [15], the author proposes a method to find the hyperparameter tuning for a deep neural network by using a univariate dynamic encoding algorithm. Work [16] presents a hyperparameter tuning framework that works on a small subset of training data using Bayesian optimization, and then it leverages the insights from the learning theory to seek more complex models by using directional derivatives. In [17] the authors propose MonkeyKing, a system that exploits past experience and collects new information to adjust parameter configurations of big data platforms. It can recommend key parameters and combine deep reinforcement learning (DRL) to optimize these key parameters to improve performance. Work [18] presents Hyperband, an algorithm for hyperparameter optimization speeding up random search through adaptive resource allocation and early-stopping. The problem is formulated as a pure-exploration non-stochastic infinite-armed bandit problem. Finally, ref. [19] proposes an approach named Auto-CASH to select both the algorithm and the hyperparameters, learning them from prior experience by means of a reinforcement learning strategy, so as to be less dependent on human expertise. In this work, we propose an approach for the automatic categorization of the tickets received by the Contact Center of the Italian National Statistics Institute (Istat) for assistance to users involved in surveys (e.g., Census of Population, Research, and Development Business survey, etc.). This Contact Center has been in operation since January 2016, and currently manages about 60 large statistical investigations and processes an average volume of about 80,000 requests per year. An efficient and accurate automatic categorization of these tickets is particularly needed. The proposed approach is composed of an initial Text Mining phase, which includes tokenization, stop-word elimination, Lemmatization with Part-Of-Speech recognition, and finally feature extraction by using Word2 vec [20]. The tickets are written in the Italian language, however, the above text processing is based on the use of a dictionary. Hence, the language can be easily modified by simply changing the dictionary. After this, our approach contains a classification phase, in which we use both deep and traditional learning. In particular, we use Convolutional Neural Networks (CNN) [21] and Support Vector Machines (SVMs) [22]. Our approach is formal and data driven: all the relevant information is extracted from the data, except for the class labels of the training set which are of course externally defined.