# Information Retrieval and Machine Learning Methods for Academic Expert Finding
## 4 Experimental Results and Discussion
### Test Collections  
We used two different collections of scientific publications to test the capabilities of the different methods. On the left-hand side of Figure 1, we present a summary of the datasets used and outline the procedure employed for partitioning the data into training and test sets. PMSC-UGR [20] is a collection of journal papers extracted from PubMed and Scopus focusing on the biomedical domain. One advantage of this collection is that the authors' names are disambiguated using their ORCID codes. Therefore, we avoid the issue of confusing publications between authors who share the same name. We specifically selected articles published between 2007 and 2016, utilizing the articles from 2007 to 2015 for the training set and those published in 2016 for the test set. Additionally, we only considered authors who had a minimum of 10 articles in the training set, ensuring a sufficient amount of information to train the models, and at least 1 article in the test set. As a result, this document collection consisted of 341,520 articles, with 304,596 articles in the training set and 36,924 articles in the test set, authored by 9453 authors. The reason for this splitting was to prevent prediction of historical events based on future information; i.e., to refrain from suggesting potential experts based on the content of a test article by relying on information from articles published after the test article's date. To ensure this, the test articles, which represented the sought-after expertise, had to have a publication date later than the training articles used to form the experts' subprofiles. This criterion guided the division of the data into training and test sets. The other collection was the COVID-19 open research dataset, CORD-19 [21], a resource for scholarly articles about COVID-19 and related coronaviruses ([[https://www.kaggle.com/datasets/allen-institute-for-ai/CORD-19-research-challenge](https://www.kaggle.com/datasets/allen-institute-for-ai/CORD-19-research-challenge)]([https://www.kaggle.com/datasets/allen-institute-for-ai/CORD-19-research-challenge](https://www.kaggle.com/datasets/allen-institute-for-ai/CORD-19-research-challenge)), accessed 19 June 2023). In this dataset, authors' names are not disambiguated and we performed some basic procedures to clean data and deal with similar author names. To mitigate name ambiguity, the initial step involved filtering out entries in which authors had provided only their initials instead of their preferred canonical full names. For instance, in cases like \"[PERSON]\", \"[PERSON]\", and \"[PERSON]\", only \"[PERSON]\" was recognized as a valid author. Furthermore, for enhanced reliability, we limited our analysis to authors with a minimum of 20 publications signed using their canonical full names, obtaining 7389 authors and 130,477 articles. We aimed to achieve a split between training and test partitions based on publication dates, following a similar approach as used for the PMSC-UGR collection. However, the concentration of articles within a brief time period posed a challenge for reli  
Figure 1: Graphical scheme of the experimental design: datasets and their partitions and ML- and IR-based recommendation and evaluation. able date-based division. Additionally, editorial factors complicated the determination of whether an article published on a certain date genuinely followed another article published slightly earlier. As a result, we chose an alternative method for splitting: an 80/20 stratified sampling approach, obtaining 104,382 articles for training and 26,095 for test. Additionally, we guaranteed that a paper co-authored by multiple authors was not included in both the train and test splits simultaneously. In Figure 2, we show two histograms that display the frequency for authors based on the number of published papers they had in each respective dataset. Despite employing distinct criteria concerning the time span and topics during the compilation of these collections, both exhibited a notable power-law pattern.