# Best Practices of Convolutional Neural Networks for Question Classification
## 3 Results and Discussion
### Learning Procedure  
The first hyperparameter defining the learning procedure considered here was the batch size, i.e., the number of examples considered together to calculate the loss value. Given the training dataset, the variability range was _batch_ \(\in\) [1,900], and all the orders of magnitude were considered, i.e., _batch_ = {1,10,100,900}. Results are reported in Table 13 and graphically represented in logarithmic scale in Figure 8.  
Figure 8 clearly shows that, while the influence of batch size was not significant for sizes between 1 and 10, as the batch size increases, the accuracy clearly decreases, with a very strong effect of this hyperparameter. This finding confirms the usefulness of employing batches instead of summing up the loss function for all the examples. In particular, the smaller the batch size was, the higher accuracy was obtained. However, one should also take into account that smaller batch sizes also cause much longer training time. For this reason, here, in order to choose the best batch size, between 1 and 10, having comparable performances, _batch_ = 10 was chosen. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multicolumn{2}{c}{_batch_} & **1** & **10** & **100** & **900** \\ \hline \multirow{2}{*}{_Acc (\%)_} & **English** & 88.1 & 88.1 & 85.1 & 80.2 \\  & **Italian** & 86.4 & 86.6 & 83.3 & 80.4 \\ \hline \hline \end{tabular}
\end{table} 
Table 13: Testing accuracy obtained by different batch sizes 1. Figure 8: Testing accuracy obtained by different batch sizes, for English (**red**) and Italian (**blue**). The dropout was also varied, in the range \(P_{keep}\) = (0,1], and in particular, in correspondence of the following representative levels: \(P_{keep}\) = {0.1,0.5,0.9,1}. Results are reported in Table 14.  
From Table 14, it can be seen that, excepting the case \(P_{keep}\) = 0.1, which causes significant accuracy worsening, the other cases were very similar. This means that for this kind of system, and for the considered size of the dataset, dropout was not strictly necessary. Therefore, for the considered dataset, the dropout can be avoided, by choosing \(P_{keep}\) = 1.0, or equivalently \(P_{keep}\) = 0.9 can be chosen. The regularization term \(l2\) was also considered, at the following levels: \(l2\) = {1.0,3.0,5.0}. Results are reported in Table 15.  
From Table 15, a slight but significant decreasing trend of the accuracy can be detected while \(l2\) increases. Therefore, \(l2\) = 1.0 was chosen. Finally, the updating rule _optimizer_ used to perform weights update by SGD backpropagation algorithm, and the associated learning rate \(\eta\), were studied together, in order to evaluate also their probable interactions. According to a full factorial design, all the combinations of factors levels were experimented, i.e., all the available updating rules _optimizer_ = {_Adadelta_,_Adagrad_,_Adam_,_Ftrl_, _GradientDescent_,_Momentum_,_ProximalAdagrad_,_ProximalGradientDescent_,_RMSProp_} combined with all the magnitude orders in the considered range of the learning rate \(\eta\)= {0.01,0.1,1,10}. Results were reported in Table 16 and shown in Figure 9. In Table 16, some results were not reported (\"-\"), since the corresponding experiments were not performed, because they make no sense in light of the other experiments. Moreover, the results of some experiments were reported as \"<20.0\", since in those cases the learning procedure did not offer acceptable accuracy. Some others were reported as \"DIV\", since the learning procedure gave exceptions due to overflow. The results indicated by an asterisk were obtained by 20,000 epochs instead of 2000; however, results with different numbers of epochs can be compared, because in all the cases the training was stopped after that convergence was reached. For each set of experiments with different learning rates, the best result is reported in bold. From Table 16, it can be evinced that some updating rules does not work with a too high learning rate, giving overflow problems.