# A Purely Entity-Based Semantic Search Approach for Document Retrieval
## 3 Materials and Methods
### Retrieval and Ranking Method
#### 3.3.3 Algorithm
Moreover, in this experiment, the achieved score is the maximum nDCG@5 score, corresponding to only two queries. This low number of queries represents a limitation of our approach. But this limitation is caused by the comparison requirements, where we obtain that maximum nDCG@5 score after selecting the queries with the highest representation (with the highest annotation average score). Thus, this is how we compared our approach to any other document retrieval approach. Table 11 shows how the LongP (Longformer) model outperforms the PES4 IR and Galago methods, with a big difference for the first three classes of queries. However, with the queries of the last class (queries whose entities have an average annotation score greater than or equal to 0.75), the LongP (Longformer) model and Galago (Dirichlet) are outperformed by our PES4 IR approach. It is important to note that for the queries of the last class, Galago (Dirichlet) outperforms the LongP (Longformer) model. While a strong approach has a good performance for the entire query set, another approach which is not as strong might offer an even better performance for some of those queries. This is one of the motivations of our approach. ### PES4 IR Tested by DBpedia Spotlight

As we mentioned before, the DBpedia Spotlight entity linker tends to assign higher scores for annotations than the REL entity linking method, which is the reason why the \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{4}{c}{**nDCG@5**} \\ \cline{2-5}  & **AVGs \(\geq\) Min** & **AVGs \(\geq\) 0.65** & **AVGs \(\geq\) 0.7** & **AVGs = 0.75** \\  & **(12 Queries)** & **(9 Queries)** & **(4 Queries)** & **(2 Queries)** \\ \hline Galago (Dirichlet) & 0.4976 & 0.5097 & 0.5463 & 0.9152 \\ LongP (Longformer) & **0.7122** & **0.7085** & **0.7769** & 0.8962 \\ PES4 IR & 0.3336 & 0.4071 & 0.4781 & **1.0000** \\ \hline \hline \end{tabular}
\end{table} 
Table 11: nDCG@5 scores for queries’ classes given by REL on Robust collection. Figure 4: nDCG@5 retrieval scores for the groups of queries from the Robust collection, annotated by REL. group of queries with the highest representation has a high value. Table 12 contains highly represented queries whose entities have an average annotation score equal to 1.0. These queries are the queries with the highest representation according to the DBpedia Spotlight entity linker. For each query whose annotation average score is equal to 1.0, Table 12 shows the detected mentions with the corresponding entities and their annotation scores. Moreover, the performance of the test performed with the DBpedia Spotlight entity linking method corresponding to the nDCG@5 scores is presented in Table 13.

As one can see in Table 12, PES4 IR with the queries annotated by DBpedia Spotlight achieves an added value for Galago with the higher represented queries. However, it is outperformed by LongP (Longformer) model for all the query classes. Thus, when PES4 IR is tested on the Robust04 collection by the queries annotated by the DBpedia Spotlight tool, it does not outperform the LongP (Longformer) model. ### Query Annotation Weakness

The weakness of the annotation of a given query could be represented by using the DBpedia spotlight, REL, and TagMe tools for the query sets such as TREC 2019 and TREC 2020 of the MSMACO collection. In the following table (Table 14), we provide a weakness point of a purely entity-based approach, which is caused by the query annotation process. We note that REL gives its annotations of this query as the highest-scoring annotated query, with an annotation score of 0.94. Such an annotation would surely negatively affect any purely entity-based approach. Generally, entity linking methods use a disambiguation process to select an entity among many candidate entities, as we explain in Section 3.1.4. In this case, for the given query \"who is [PERSON]\", among many entities (known people, with the same name) the REL method selects \"[PERSON]_(poet)\" to be the entity.