## 4 Results  
### Performance Evaluation  
In this experiment (a) we investigate whether the predicted classifications have any correlation with human annotation, (b) we compare the proposed classification mechanism against the baseline classification models for small noisy and formal datasets respectively, (c) we compare two different ways of generating the embedding layer, and (d) we test two different options of validation methods. Table 2 presents the classification results (Precision and Recall) for the different MT outputs over the two different datasets. The \(C1\) corpus presents a classification increase, for both language pairs (accuracy: 72% EN-EL/70% for EN-IT), in contrast to the \(C2\) corpus (accuracy: 68% for EN-EL/65% for EN-IT), even though the \(C1\) corpus contains a lot of noise. This is probably due to the fact that the \(C1\) corpus contains more sentences, and, also, because the \(C2\) corpus has richer vocabulary and more formal structure. It is more difficult for the classifier to choose the best MT output, because the SMT output is more similar to the NMT output in this corpus (\(C2\)). It is also observed that both evaluation metrics chose the NMT model over the SMT one, which is in accordance to the annotators' results. In addition, the aforementioned accuracy results are obtained when the NN uses the simple embedding layer. However, when the pre-trained embeddings are used, the model does not lead to better results (average accuracy of \(C1\) and \(C2\): 66% for EN-EL/65% for EN-IT), since the embeddings are trained on the general-purpose corpus, which is not representative of the input corpora used therein. At this point, it is worth mentioning that the pre-trained embeddings seem to be more effective for the EN-IT pair than for the EN-EL language pair. As far as the different types of corpora are concerned, pre-trained embeddings are more efficient for the \(C2\) corpus (average accuracy of EN-EL and EN-IT: 66%) than the \(C1\) corpus (average accuracy of EN-EL and EN-IT:  
Figure 1: Neural network (NN) architecture. 64%). This is probably due to the fact that the C2 corpus has richer vocabulary than the C1 corpus. An approach to improve the classification accuracy of a small and noisy dataset is to apply the SMOTE oversampling technique on the training data. Using SMOTE, the sentences of the minority class (SMT) doubled in number, and the total number of sentences reached 3024 for _C1_ and 2276 for C2. It is important to compare the performance between the 82 and the 86 feature dimensions, with and without the SMOTE filter. When SMOTE is applied, a small accuracy increase is observed on the 82 features (average accuracy of _C1_ and _C2_: 68% for EN-EL/67% For EN-IT), and an even higher increase on the 86 features (average accuracy of _C1_ and _C2_: 70% for EN-EL/68% for EN-IT). It is interesting that the EN-EL corpora outperformed EN-IT in all the experiments. The results with the use of the two new suggested features are generally better for both corpora and language pairs. Firstly, k-fold _CV_ was used, which is a reliable method for testing the models, and a value of \(k\) = 10 is very common in the field of machine learning [44] (Table 2). Secondly, part of the data (70%) is kept for training, and part (30%) is applied for testing (Table 3). Given that both classes are of interest, the symmetric Matthews correlation coefficient (_MCC_) metric [45] (a special case of the \(\phi\) phi coefficient [46]) is used, as it constitutes a good way to describe the relation of _TP_ (true positive), _FP_ (false positive) and _FN_ (false negative) values by a single number.