# An Algorithm for the Detection of Hidden Propaganda in Mixed-Code Text over the Internet +
## 5 Evaluation and Results Discussion  
In the previous section, the proposed algorithm for supporting the analysis of mixed-code text has been described, as well as the approach for assessing and selecting the most promising machine learning classifier for the propaganda detection step has been contextually presented. This section, instead, aims to discuss and show how the evaluation of the overall workflow has been conducted by explaining the encountered problem, by clarifying how the experiment has been setup, as well as by discussing the achieved results. One of the problems encountered during the evaluation part concerned the lack of available dataset containing word art mixed code text and, especially, related to propaganda. To deal with it, the dataset described in Section 4.3 [37] has been taken in consideration as a starting point. In particular, a subset, called \(SS\), of its instances has been selected by creating a smaller balanced dataset, that means by taking 50% of sentences labelled as propaganda and 50% of them labelled as nonpropaganda related. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Trained Classifier** & **Accuracy** & **Precision** & **Recall** & **F1-Score** \\ \hline Multinomial Naive Bayes count\_vectorizer & 0.74 & 0 & 0.74 & 0.74 \\ \hline Multinomial Naive Bayes & & & & \\ tfid\_vectorizer\_word & 0.71 & 0.70 & 0.70 & 0.70 \\ \hline Multinomial Naive Bayes & & & & \\ tfid\_vectorizer\_word\_ngram & 0.68 & 0.68 & 0.66 & 0.65 \\ \hline Multinomial Naive Bayes & & & & \\ tfid\_vectorizer\_ngram\_chars & 0.68 & 0.68 & 0.67 & 0.68 \\ \hline Support Vector Machine count\_vectorizer & 0.80 & 0.80 & 0.80 & 0.80 \\ \hline Support Vector Machine & & & & \\ tfid\_vectorizer\_word & 0.82 & 0.82 & 0.82 & 0.82 \\ \hline Support Vector Machine & & & & \\ tfid\_vectorizer\_word\_ngram & 0.72 & 0.72 & 0.70 & 0.71 \\ \hline Support Vector Machine & & & & \\ tfid\_vectorizer\_ngram\_chars & 0.82 & 0.82 & 0.82 & 0.82 \\ \hline Logistic Regression count\_vectorizer & 0.79 & 0.78 & 0.79 & 0.78 \\ \hline Logistic Regression tfid\_vectorizer\_word & 0.73 & 0.72 & 0.72 & 0.72 \\ \hline Logistic Regression & & & & \\ tfid\_vectorizer\_word\_ngram & & & & \\ \hline Logistic Regression & & & & \\ tfid\_vectorizer\_ngram\_chars & 0.71 & 0.71 & 0.71 & 0.71 \\ \hline Convolutional Neural Network & 0.94 & 0.92 & 0.92 & 0.91 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Comparison of the performance achieved from each classifier with oversampling. After that, an online tool, called Universal Leet [53] has been used. Given in input to such tool a word \(W\), it is able to automatically generate a possible related _Art Form Word_ (\(W_{AFW}\)). As a consequence, the experiment dataset has been built by replacing in each sentence of the called \(SS\) subset, at least one word or even the full text with its related art form word, automatically generated from the above mentioned tool, so as to obtain a _Art Form Word_ dataset (\(SS_{AFW}\)) with labels. In particular, at first we used alternatively three standard available modalities: _basic_, _advanced_, and _ultimate_ feet to obtain the first version, and then in order to make such versions nonstandard but more human-like in terms of their variety, we have further updated part of them manually, so as to make them more difficult and less machine-related. All this encoding step has been done manually, because based on our knowledge we have not found any APIs, which allowed us to automate this process. An excerpt is shown in Table 7.  
Instead, the WordNet package [54] has been used as reference English dictionary, in order to implement the _Word Selection step_ of the proposed normalisation algorithm, described in Section 4.2. It is centred on the Python's Natural Language Toolkit (NLTK) module and it consists of a database where the collected nouns, adjectives, adverbs and verbs are grouped into a set of cognitive synonyms, which are called synsets.