## 3 Results
### SME Use Cases
In contrast, the MAP score of the BERT-based method increases to its maximum at the 11 th rank with a MAP score of 0.625 and then gradually declines as well while continuously yielding a better MAP score than the MeSH-based method. The aggregated overall results for the test cases with the SMEs are visualized in Figure 7. Due to a lack of MeSH terms in clinical trials, a second search was not conducted; hence, only the top 10 clinical trials for both SMEs of the first search stage are compared. Figure 6: The goodness of ranking is illustrated as a line chart depicting the MAP score at a given document rank of the MeSH- and BERT-based method for the top 50 publications averaged for both SME use cases. Figure 7: Averaged precision of both search methods overall, for publications and for clinical trials of both SME use cases. For each category, n defines the number of total documents per bar. The BERT-based method performed better overall by retrieving proportionally more relevant documents. While the overall precision, e.g., how many retrieved documents were relevant, for the BERT-based method with 44 relevant documents was 73.3%, the precision for the MeSH-based method with 20 relevant documents was only 33.3%. The BERT-based method performed similarly well on publications with a precision of 72.5% and 29 retrieved relevant publications as when searching for clinical trials with a precision of 75% and 15 retrieved relevant clinical trials. The MeSH-based method yielded a lower performance and retrieved 18 relevant publications with a precision of 45% and retrieved only 2 relevant clinical trials with a precision of 10%. Besides the notable difference in overall performance between both methods, a contrast between precision when retrieving publications compared to clinical trials is evident for the MeSH-based method. To better understand both search methods regarding the retrieved documents, result sets were compared regarding the number of the same retrieved documents and the same MeSH terms of retrieved documents. To analyze the similarity of the result sets of both methods, the Jaccard index of retrieved publications with respective PubMed identifiers (PMID) was investigated. Figure 8 visualizes the Jaccard index of documents in the top 500 results of both search methods for the first and second SME use cases. For the SME A use case (a), the maximum Jaccard index is 0.013, indicating that the result sets are almost disjoint. In the SME B use case (b) the maximum Jaccard index is 0.11, indicating that the result sets of both search methods are only slightly similar. Moreover, a slight increase in similarity for the SME A use case and a stronger increase in the similarity for the SME B use case can be observed, indicating that the result sets might converge later on. The content of the retrieved publications as expressed in the form of the respectively assigned MeSH terms was compared with the result set of both methods and the positive rated MeSH terms by the SME. The number of unique MeSH terms occurring in the top 500 retrieved publications and the overlap between both search methods with the SME-rated relevant terms is visualized in Figure 9.  
The MeSH-based method features more unique MeSH terms compared to the BERT-based method and a strong overlap between the MeSH terms of the result sets with respective Jaccard indices of 38.7% for SME A and 30.6% for SME B exists. Moreover, it is noticeable that almost all SME-rated relevant terms, namely 87.2% for SME A and 88.2% for SME B occur in either result set. Only a few selected SME-rated relevant terms are present in one result set and not the other. For the first SME use case, all terms rated as relevant were present in the result sets and for the second SME use case, only three terms were not present in any retrieved publication. Figure 8: Diversity of result sets from both search methods on document level (PMID) measured by the Jaccard index, i.e., retrieved publications, by both search methods for both SME use cases in the first 500 results.