## 4 Methodology
### Hyperparameter Tuning  
The NLU problem is considered an AI-hard problem (meaning that the created software should be as intelligent as a human), and a lot of effort has been put into the optimization of DNN methods (as, e.g., in [41]). DNN methods have many hyperparameters, and each hyperparameter may have several determined choices (e.g., several types of activation functions) and discrete numeric (e.g., number of neurons) or real numeric (e.g., dropout from an interval [0, 1]) values. However, choosing optimal hyperparameter values manually is a difficult task, even for human experts. To overcome this problem, an open-source Python's library, Hyperas (the information about Hyperas is in [[https://github.com/maxpumperla/hyperas](https://github.com/maxpumperla/hyperas)]([https://github.com/maxpumperla/hyperas](https://github.com/maxpumperla/hyperas))), implemented to optimize hyperparameters in Keras models automatically, has been used. The following options were experimentally investigated:  
* Several DNN architectures (shallower and deeper), having different numbers of hidden layers (i.e., series of convolutional layers in CNN, simple or stacked LSTM and BiLSTM versions). * DNN hyperparameters: numbers of neurons (100, 200, 300, or 400), dropouts (values from an interval [0, 1]), recurrent dropouts (from [0, 1]), activation functions (relu, softmax, tanh, elu or selu), optimizers (Adam, SGD, RMSprop, Adagrad, Adadelta, Adamax or Nadam), batch sizes (32 or 64), and numbers of epochs (20, 30, 40 or 50). Tuning of the DNN models (i.e., their hyperparameters) was performed automatically. The training of some models was done on the training split (which contains 80% of instances from the shuffled training set), and the validation was done on the rest (20% of instances from the training set). The hyperparameter optimization was done to increase the validation accuracy, and, for this reason, the following two optimization algorithms were used:  
* Random.suggest performs a random search over a set of hyperparameter values in 100 iterations (the experimental investigation revealed that 100 iterations are enough to find the optimal hyperparameter value set that gives maximum accuracy on the validation dataset). When seeking the most accurate combination, it explores the hyperparameter value space by randomly checking different combinations. * Tpe.suggest (tree-structured Parzen estimator) [42] performs a Bayesian-based iterative search (for the schematic representation of TPE, see Figure 1). The search strategy of TPE contains two phases. During an initial warm-up phase, it randomly explores a hyperparameter value space. These hyperparameter values can be conditional (additional layer in the architecture), sampled from an interval (as, e.g., for a dropout), or chosen from a determined list of values (e.g., activation functions). The chosen hyperparameter value combinations are used to train a model (with the training dataset split), which is evaluated with the validation split to see each chosen hyperparameter value combination impact on the accuracy. The warm-up phase lasts for _n_init_ iterations (_n_init_ = 20 in our experiments) and builds a function based on the Bayesian rule presented in Equation (1). \[P(acclparam)=\frac{P(param|acc)\times P(param)}{P(acc)}\] (1)  
_P(acclparam)_ is the probability of some validation accuracy (\(acc\)) to be achieved with a determined set of hyperparameter values (\(param\)). Based on this accuracy, hyperparameter value combinations are distributed into good and bad splits. The parameter \(\gamma\) allows us to determine the size of a good split. In our experiments, \(\gamma\) = 0.25, which means that 25% of all hyperparameter value combinations belong to a good split, and the rest (75%) belong to a bad split. Based on how hyperparameter value  
Figure 1: A schematic representation of the TPE (tree-structured Parzen estimator) optimization algorithm. combinations are distributed, the accuracy threshold (denoted as \(acc^{\prime}\)) is calculated.