# Linked Data Triples Enhance Document Relevance Classification
## 6 Experimental Description
### Effect of Document Length on BERT Models  
BERT outperforms other models by representing contextualized and self-attentive word vectors. The pre-training of BERT teaches the model to understand language structure  
\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Feature Set** & **Accuracy** & **Macro-F1** & **Weighted-F1** \\ \hline Text(GloVe) & 49.7 & 48.5 & 49.3 \\ +Topics \(\dagger\) & 52.1 & 50.8 & 51.8 \\ +NEs \(\dagger\) & 54.2 & 52.9 & 53.6 \\ +Triples & 50.4 & 49.7 & 50.6 \\ +Topics+NEs \(\dagger\) & 55.0 & 54.3 & 55.2 \\ +Topics+Triples \(\dagger\) & 52.5 & 51.4 & 52.4 \\ +NEs+Triples \(\dagger\) & 54.7 & 53.2 & 54.1 \\ +Topics+NEs+Triples \(\dagger\) & **56.1** & **55.0** & **55.9** \\ \hline Text (SEs) & 74.1 & 73.4 & 74.2 \\ +Topics \(\dagger\) & **74.8** & **73.9** & **74.8** \\ +NEs \(\dagger\) & 59.5 & 58.9 & 59.7 \\ +Triples (SEs) & 74.1 & 72.9 & 73.9 \\ +Topics+NEs \(\dagger\) & 62.2 & 61.3 & 62.1 \\ +Topics+Triples (SEs) \(\dagger\) & 74.6 & 73.9 & 74.8 \\ +NEs+Triples (SEs) \(\dagger\) & 59.1 & 58.8 & 59.5 \\ +Topics+NEs+Triples (SEs) \(\dagger\) & 61.8 & 60.7 & 61.6 \\ \hline Text (BERT) & 83.3 & 82.6 & 83.3 \\ +Topics \(\dagger\) & **84.3** & **83.6** & **84.4** \\ +NEs \(\dagger\) & 82.4 & 81.5 & 82.3 \\ +Triples (BERT) & 83.9 & 83.1 & 84.0 \\ +Topics+NEs & 83.8 & 83.1 & 83.8 \\ +Topics+Triples (BERT) & 82.2 & 81.5 & 82.3 \\ +NEs+Triples (BERT) & 83.3 & 82.3 & 83.1 \\ +Topics+NEs+Triples (BERT) & 83.5 & 82.4 & 83.3 \\ \hline \hline \end{tabular}
\end{table}
Table 4: News-feed classification results on the 20 News groups dataset. \(\dagger\)s indicate statistically significant differences (using McNemar’s test) with reference to the baseline models. Note: SEs = Sentence embeddings, NEs = Named entities.  
Figure 8: Performance results of classifiers from different inputs on the 20 News groups dataset. The first block of results uses GloVe-based representations. The second block is models using the InferSent sentence encoder. The third block uses BERT-based models. \(*\) indicate the best model from each class.  
by learning linguistic features through its projections. A study by [PERSON] [43] finds that each attention head in BERT attends to a different set of features. Though the features are not explicitly defined, the attention heads of BERT extract syntactic relations of subject-verb-object. This makes the triple structure redundant when added to the baseline text model and accounts for the absence of any significant performance gain.  
BERT also restricts its input vector length to 512 tokens, so important information from long documents is truncated. This impacts model performance. An interesting finding from our experiments is that combining \"Triples\" with \"Text\" input results in significant performance improvement on long documents. This is because the text model of BERT is restricted to the first 512 tokens, so adding Triples enriches the model beyond the plain text. Triples extracted from document sentences are mostly shorter than plain text and, therefore, provide a valuable abbreviated input to the BERT model.  
To validate this claim, long documents (those with more than 600 words) are considered from Energy-Hub and the 20 News groups dataset. The Reuters dataset has very few long documents, thus being unsuited for this comparison.  
The results from Table 5 show that classification performance on long documents is improved for the Energy Hub and standard 20 News groups datasets. [PERSON]'s tests are conducted on these models, and it is found that long documents produce significantly different results. This provides further evidence to our claim that triples constructed using topics and named entities contain significant latent semantic information and can be used to enrich document representations.