# Linked Data Triples Enhance Document Relevance Classification
## 6 Experimental Description
### Experiments on Energy Hub
#### 6.2.2 Results  
We used accuracy (the number of observations correctly made), precision (the percentage of relevant documents in those retrieved), recall (the percentage of documents in the retrieval set compared to number known to be relevant), F1 score (harmonic mean between precision and recall), and AUC (Area Under Curve) as evaluation metrics reported in Table 2 and visualised in Figure 6. When comparing models, accuracy is not the ideal metric due to the risk of model bias towards a specific class, hence F1-scores being computed. Consider the second block of results in Table 2. These show the performance of the additional features compared to a baseline model consisting of Text (with sentence embeddings), shown in the table as Text (SEs). The sentence embeddings-based model performs better than GloVe-based models with sentence vector representations of text achieving 67.8% accuracy and 70.3% F1 score. The \"Text (SEs)+topics\", \"Text (SEs)+triples\", and \"Text (SEs)+topics+triples\" models perform better than the baseline \"Text (SEs)\" model. \begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Feature Set** & **Accuracy** & **Precision** & **Recall** & **F1 Score** & **AUC** \\ \hline Text (with GloVe) & 64.5 & 82.3 & 49.8 & 62.1 & 76.0 \\ +Topics & 64.0 & 77.9 & 53.3 & 63.3 & 72.7 \\ +NEs & 66.2 & 71.0 & 71.0 & 71.0 & 73.9 \\ +Triples \(\dagger\) & **68.9** & **77.8** & 65.3 & 71.0 & **76.9** \\ +Topics+NEs & 66.5 & 71.3 & 71.3 & 71.3 & 74.2 \\ +Topics+Triples & 68.0 & 72.7 & 72.2 & 72.5 & 74.8 \\ +NEs+Triples & 60.5 & 76.0 & 47.0 & 58.1 & 69.1 \\ +Topics+NEs+Triples & 67.6 & 67.0 & **87.7** & **76.0** & 73.9 \\ \hline Text (SEs) & 67.8 & 76.1 & 65.3 & 70.3 & 76.1 \\ +Topics \(\dagger\) & 70.4 & **81.2** & 64.0 & 71.6 & 78.4 \\ +NEs & 63.8 & 71.3 & 63.4 & 67.1 & 69.2 \\ +Triples (SEs) & 70.6 & 79.8 & 66.2 & 72.4 & 77.8 \\ +Topics+NEs & 65.1 & 70.3 & 69.4 & 69.8 & 70.2 \\ +Topics+Triples (SEs) \(\dagger\) & **73.2** & 78.4 & **74.4** & **76.4** & **79.3** \\ +NEs+Triples (SEs) & 62.3 & 69.7 & 62.5 & 65.9 & 68.2 \\ +Topics+NEs+Triples (SEs) & 63.6 & 68.1 & 70.7 & 69.3 & 68.4 \\ \hline Text (BERT) & 71.3 & **88.5** & 58.3 & 70.3 & 84.3 \\ +Topics \(\dagger\) & 75.1 & 85.8 & 68.7 & 76.3 & 86.1 \\ +NEs \(\dagger\) & 76.4 & 85.6 & 71.6 & 78.0 & **86.8** \\ +Triples (BERT) \(\dagger\) & **77.7** & 78.0 & **86.1** & **81.8** & 86.1 \\ +Topics+NEs \(\dagger\) & 75.9 & 81.8 & 75.3 & 78.4 & 85.2 \\ +Topics+Triples (BERT) \(\dagger\) & 76.6 & 88.3 & 69.0 & 77.5 & 86.6 \\ +NEs+Triples (BERT) \(\dagger\) & 77.5 & 82.1 & 78.5 & 80.3 & 84.6 \\ +Topics+NEs+Triples(BERT) \(\dagger\) & 75.5 & 81.2 & 75.3 & 78.2 & 85.4 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: News-feed classification results on the Energy-Hub dataset. \(\dagger\)s indicate statistically significant differences (with [PERSON]’s test) with reference to the baseline model. For instance, the addition of triples improves the baseline mode (Text (with Glove)) from 64.5% to 68.9%, a difference that is statistically significant. Note: SEs = Sentence embeddings, NEs = Named entities. Comparing all sentence embedding-based models, \"Text+topics+triples\" resulted in the best \"F1\" of 76.4%. The third block of results in Table 2 uses BERT as the baseline model for presenting text. The BERT-based models show even better performance than GloVe and sentence embeddings. Considering \"Text (BERT)\" as a baseline, adding topics, named entities, and triples has positive impact by improving classification performance. Overall, BERT-based \"Text+triples\" proved to be the best model with an F1 score of 81.8%, and it performed significantly better than the baseline \"Text (BERT)\" model. The performance of a neural network is affected by initialization of weights [42]. The variation in performance of a few proposed models is very small compared to the baselines. To make reliable conclusions, the classification results of each model at document level is compared to its baselines.