# A Purely Entity-Based Semantic Search Approach for Document Retrieval
## 3 Materials and Methods
### Retrieval and Ranking Method
#### 3.3.3 Algorithm
Moreover, for Galago, the added value is expressed by all the used metrics. Furthermore, PES4 IR provides an added value for any state-of-the-art method used on the TREC 2004 Robust collection, such as when one uses PES4 IR for the highly annotated queries (AVG_score \(\geq\) 0.75, annotated by REL) and any other state-of-the-art method for the rest of the queries. In this case, an added value is offered by PES54 IR unless the SOTA method alone can reach the maximum nDCG@5 score for those highly annotated queries. In the discussion session (see Section 5.1), we give more details about the added value offered by PES4 IR. #### 4.3.2 PES54 IR with LongP (Longformer) Model

We compared PES54 IR with the LongP (Longformer) model and combined them to have a better performance for the ad hoc document retrieval task. Moreover, in this experiment, PES54 IR was used for the set of queries with the highest representation (AVG_score \(\geq\) 0.75, annotated by REL), and LongP (Longformer) was used for the rest of the queries. The added value achieved by PES54 IR appeared clearly when it was used alongside the LongP model. The results are illustrated in Table 8.

In this table, the evaluation is given by the nDCG@5 metric, where it shows a better ranking performance; this is due to the outperformance of PES54 IR upon the LongP (Longformer) model for the queries with the highest annotation. In Section 5.1, we provide the details of the added value achieved by PES54 IR. ### Results of the Experiment on MSMARCO

We would have tested PES54 IR by queries annotated by both the REL and the DBpedia Spotlight entity linking methods; however, we tested it only with the DBpedia Spotlight tool (see Appendices B.3 and B.4). The reason for not testing PES54 IR with queries annotated by the REL tool is that among the completely annotated queries, of the query sets of both TREC-DL-2019 and TREC-DL-2020, there is no query whose annotation avg score is greater than 0.75 (for the TREC-DL-2020 query set, see Appendix B.1); and for the query set of TREC-DL-2019, there is only one query, but it has a scoring issue (see Appendix B.2). In the discussion section (see Section 5.3), we provide the details related to that scoring issue. In the following table (Table 9), we provide the performance and results of our approach (PES54 IR) and of the LongP (Longformer) model on the MSMARCO collection. Moreover, the nDCG@20 metric is used. It is important to note that the Python API of the MSMARCO collection does not provide judgment values (qrels) for some queries of the TREC DL 2019 and TREC DL 2020 query subsets. Among them, the queries with the highest annotation exist (annotated by the DBpedia Spotlight tool, whose annotation avg score is equal to 1), and with the queries with the highest annotation, PES54 IR is supposed to achieve its best performance. However, the corresponding judgment values are provided by the MSMARCO collection. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Method** & **nDCG@5** & **MAP** & **P@5** \\ \hline Galago (Dirichlet) & 0.3729 & 0.1534 & 0.3855 \\ Galago + PES54 IR & **0.3758** & **0.1540** & **0.3855** \\ \hline \hline \end{tabular}
\end{table} 
Table 7: PES54 IR’s added value with Galago. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Method** & **nDCG@5** & **MAP** & **P@5** \\ \hline LongP (Longformer) & 0.6542 & **0.3505** & 0.6723 \\ LongP (Longformer) + PES54 IR & **0.6551** & 0.3492 & **0.6731** \\ \hline \hline \end{tabular}
\end{table} 
Table 8: PES54 IR and LongP (Longformer) on Robust Collection. The LongP (Longformer) model outperforms PESS4 IR for the first two classes for each set (TREC DL 2019 and TREC DL 2020). And for the last class, where PESS4 IR is supposed to achieve its best performance with the queries with the highest annotation, there are no corresponding qrels; this why we put \"None\".