## 4 Results
### Model Evaluation and Validation
#### 4.2.2 CNN Classifier Results  
The hyper-parameters used to train the CNN model are shown in Table 5. The results obtained when using both Gensim and FastText generated embeddings were studied, and the \(F_{1}\) obtained was similar when using or not using pre-trained word-embeddings. Only the number of dimensions and epochs had a great impact on the performance and the computational cost of the model.  
With regard to the batch size, sizes of 1, 10, 64, 128, and 512 were investigated and, as expected, the higher the value, the greater the computational efficiency. The noisiness of the gradient estimate was reduced in batch sizes by using higher values. This can be explained by the fact that updating by one single sample is noisy when the sample is not a good representation of all the data. We should consider a batch with a size that is representative of the whole dataset. For values higher than 128, the predictive performance deteriorated in earlier epochs during training and, therefore, we chose a value of 128. In fact, it has been reported that the loss function landscape of deep neural networks is such that large-batch methods are almost invariably attracted to regions with sharp minima [33] and that, unlike small-batch methods, they are unable to escape the basins of these minimizers. When using a larger batch, there is consequently a significant degradation in the quality of the model, as measured by its ability to generalize.  
\begin{table}
\begin{tabular}{l l} \hline \hline \multicolumn{1}{c}{**Hyper-Parameter**} & **Value** \\ \hline Learning rate & 0.1 \\ Size of word vectors & 100 \\ Size of the context window & 5 \\ Number of epochs & 100 \\ Minimum number of word occurences & 1 \\ Number of negatives sampled & 5 \\ Loss function & Softmax \\ Minimum length of char n-gram & 0 \\ Maximum length of char n-gram & 0 \\ Maximum length of word n-gram & 1 \\ Sampling threshold & \(10^{-4}\) \\ Rate of updates for the learning rate & 100 \\ Use of pre-trained word vectors for supervised learning & Yes \\ \hline \hline \end{tabular}
\end{table}
Table 4: FastText classifier hyper-parameters. Optimization was performed using random search [22].  
Figure 4: Learning curves with 5-fold cross-validation on the FastText classifier using as input pre-trained word-embeddings learned (**a**) with bigrams and (**b**) without bigrams. The green region represents the standard deviation of the model.  
The CNN learning curve (Figure 3b) shows that the network is capable of generalizing well and that the model is robust. Unlike that which occurs with the FastText classifier, no overfitting is produced when the dataset is small.  
Nonetheless, it also had a bias error, but, in this case, the model achieved higher scores for both the training and the validation sets, converging to a maximum \(F_{1}=0.91\), beyond which adding more data does not appear to be beneficial.  
One additional difference with the FastText learning curve is that the CNN model needs more data to learn, in comparison with FastText. This is reflected by the fact that the CNN model was underfitting and not properly learning for a sample size \(10^{3}\) with a validation score of only 0.72, while for FastText and a sample size \(10^{3}\), the model was clearly overfitting with a training score close to 1.  
The model of the whole dataset, using 3,257,179 training examples, bigrams, and pre-trained word-embeddings, eventually yielded an accuracy of 0.91 for the validation set comprising 814,295 samples. The coefficient of agreement between the predicted and the true labels in the validation set was \(\kappa=0.83\) (see Table 3), which is regarded as an almost perfect agreement and implies that the model is reliable.