# Linked Data Triples Enhance Document Relevance Classification
## 6 Experimental Description
### Experiments on Reuters and 20 News Groups
\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Feature Set** & **Accuracy** & **Precision** & **Recall** & **F1 Score** & **AUC** \\ \hline Text (with GloVe) & 86.4 & 87.8 & 78.6 & 83.0 & 93.4 \\ +Topics & 86.4 & 88.2 & 78.1 & 82.9 & 93.0 \\ +NEs & 85.8 & 88.5 & 76.0 & 81.8 & 91.5 \\ +Triples \(\dagger\) & **89.9** & **90.1** & 85.4 & **87.7** & **95.0** \\ +Topics+NEs & 86.0 & 80.8 & **87.5** & 84.0 & 94.4 \\ +Topics+Triples & 87.7 & 90.0 & **79.7** & 84.5 & 93.4 \\ +NEs+Triples & 86.0 & 82.0 & 85.4 & 83.7 & 93.8 \\ +Topics+NEs+Triples & 88.2 & 85.9 & 85.9 & 85.9 & 94.1 \\ \hline Text (SEs) & 95.8 & 94.8 & 95.3 & 95.1 & 99.4 \\ +Topics & **96.3** & 93.5 & **97.9** & **95.7** & **99.5** \\ +NEs \(\dagger\) & 89.1 & 86.6 & 87.5 & 87.0 & 94.5 \\ +Triples (SEs) & **96.3** & **96.3** & 94.8 & 95.5 & 99.3 \\ +Topics+NEs \(\dagger\) & 91.0 & 90.8 & 87.5 & 89.1 & 95.8 \\ +Topics+Triples (SEs) & **96.3** & 94.9 & 96.4 & 95.6 & 99.3 \\ +NEs+Triples (SEs) \(\dagger\) & 90.6 & 89.8 & 87.5 & 88.7 & 96.0 \\ +Topics+NEs+Triples (SEs) \(\dagger\) & 91.5 & 92.7 & 86.5 & 89.5 & 95.9 \\ \hline Text (BERT) & 98.4 & 96.9 & **99.4** & 98.2 & **99.8** \\ +Topics & 98.2 & 97.9 & 97.9 & 97.9 & **99.8** \\ +NEs \(\dagger\) & 95.6 & 90.9 & **99.4** & 95.0 & 99.6 \\ +Triples (BERT) & 97.5 & **99.4** & 94.7 & 97.0 & 99.7 \\ +Topics+NEs & 97.3 & 96.8 & 96.8 & 96.8 & 99.7 \\ +Topics+Triples (BERT) & 98.0 & 98.9 & 96.3 & 97.6 & **99.8** \\ +NEs+Triples & **98.9** & **99.4** & 97.9 & **98.6** & **99.8** \\ +Topics+NEs+Triples (BERT) & 98.0 & 98.4 & 96.8 & 97.6 & 99.6 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: News-feed classification results on the Reuters dataset. \(\dagger\)s indicate statistically significant differences (using [PERSON]’s test) with reference to the baseline models. Note: SEs = Sentence embeddings, NEs = Named entities. Consider the second block of results in Table 3, which shows the performance of the additional features compared to a baseline model consisting of Text (with sentence embeddings), shown in the table as Text (SEs). A similar pattern of results are observed consistent with those achieved with the Energy-Hub dataset. The addition of our proposed lexical semantic and structural features to the baseline \"Text (SEs)\" model produced better results, although the model results are not significant according to [PERSON]'s test. \"Text (SEs)+Topics\", \"Text (SEs)+Triples\", and \"Text (SEs)+Topics+Triples\" show marginal improvements over the baseline \"Text (SEs)\" model. BERT models are the best performing ones with each model producing less than 10 wrongly classified documents. \"Text (BERT) + NEs + Triples\" is the best performing model with an F1 score of 98.6% compared to 98.2% over the baseline BERT. The 20 News groups is a multi-class dataset, so metrics, like Macro F-score (average F-score for each class) and Weighted F-score (average F-score considering class proportions), are used in addition to accuracy. Similar to the earlier experiments, adding proposed features to the plain text (baseline) (\"Text (GloVe)\") increased the classification metrics, with the best being \"Text (GloVe)+Topics+NEs+Triples\" in Table 4. On sentence embedding and BERT models, \"Text+Topics\" achieved the best performance, while triples delivered only slight improvements to the baseline models. Abstract LDA topics are an important feature set for classifying 20 News groups and have shown significant improvement in Figure 8, this being the precise way they are organized. To conclude, adding semantic feature-based models significantly improves performance when compared to baseline \"text\" models, although the additional features, once added, do not have a significant impact on the baseline sentence embedding or BERT models. The important aspect of the experiments on the Reuters and 20 News groups datasets is the consistency of the results with those achieved on the Energy-Hub dataset.