# Hyperparameter Black-Box Optimization to Improve the Automatic Classification of Support Tickets
## 4 Experimental Results  
The derivative-free optimization algorithm described in Section 3 to solve the black-box problem 1 has been implemented in Python, and the CNN and SVM classifiers have been realized in the same language by using keras library [29] form scikit learn [25]. We design our experiments to determine the effectiveness of an automatic approach for ticket categorization problems, given the difficulties of a real-world case, and to compare the black-box approach proposed for the optimization of the hyperparameters to a standard grid search. We consider the case of the tickets received by the Contact Center of the Italian National Statistics Institute (Istat) for assistance to users involved in surveys. This Contact Center has been in operation since January 2016, and currently manages about 60 statistical investigations and processes an average volume of about 80,000 requests per year. The ticket categorization operation is currently performed by human experts, and though it is often performed in a more coarse-grained manner, it requires a considerable amount of work. In our experiments, we have focused on the tickets arising from some economic surveys, such as the \"Research and Development Business\" survey and the \"Information and Communication Technologies usage in Enterprises\" survey. This type of survey contains many of the critical issues that users may encounter during the compilation, due to the complexity of the questionnaires. Indeed, they have the following characteristics: they require detailed quantitative information; they use tabular forms for filling out many questions; they contain a large number of prompts, many of which are blocking (also known as _hard prompts_). Therefore, this dataset is well representative of the various cases of requests for assistance. We prepared a training set of 8076 tickets, whose class has been determined by experts in the field using 7 distinct classes, which are listed below:  
1. General information: tickets that require generic information on the survey, like topic or use of the data, etc. 2. Usability: tickets relating to the difficulties in accessing the site or which highlight problems relating to the usability of the electronic questionnaire (e.g., non-editable fields, methods of sending the questionnaire, etc.) or tickets requesting the reopening of the questionnaire already sent, to proceed with the correction of data entered incorrectly. 3. Information on questions: tickets requesting assistance in completing a specific question in the questionnaire (e.g., the question is not applicable to the case of the user, or the case of the user is not present in the questionnaire). 4. Interaction with Istat: tickets that highlight a criticality in the communication process that took place between Istat and end users (e.g., requests for information on the reference year of the survey, the obligation to reply, the deadline for completion, confirmation of the sending of the questionnaire, extensions for the completion of the questionnaire). 5. Eligibility: tickets that show difficulties in understanding whether the unit has the characteristics to be part of the sample or not and therefore is actually required to participate in the survey. 6. Indeterminable / Unclassifiable: tickets with ambiguous content, or tickets with several problems highlighted at the same time, so they cannot be classified. 7. Rest: tickets describing well-defined problems not belonging to the previous classes, however, these problems have a small frequency (they appear rarely). So, instead of using several other small classes, they have been put all into one class: \"the rest of the tickets\". The class subdivision of the mentioned training set is the following: 239 in class 1, 219 in class 2, 712 in class 3, 2714 in class 4, 1200 in class 5, 418 in class 6, and 2574 in class 7. To perform the classification task, we have selected 4,038 records as a training set, that is 50% of the total dataset, and the rest has been used for testing.