## 4 Experiments
### Underlying Systems for Overlapping Clustering
As noted in Section 2.2, PubMed recommends related articles for a given article, based on an article similarity measure that has considered several kinds of indicators [23; 24]. The similarity measure was one of the best to cluster scholarly articles [19]. Following [19], for each article \(a1\), a ranked list of top-50 similar articles recommended by PubMed is retrieved. Equation (15) is then employed to estimate how an article \(a2\) is similar to \(a1\) based on the rank of \(a2\) in the ranked list (i.e., \(RankSim(a1,a2)\), see Equation (15)). \(PMS\) between two articles \(a1\) and \(a2\) is simply the average of \(RankSim(a1,a2)\) and \(RankSim(a2,a1)\), as defined in Equation (16). \[RankSim(a1,a2) = Max\{0,\ (51-Rank(a1,a2))\times 0.02\} \tag{15}\] \[PMS(a1,a2) = \frac{RankSim(a1,a2)+RankSim(a2,a1)}{2} \tag{16}\]  
_OK_ was shown to be one of the best techniques to identify related articles as well [18]. It employs Equation (4) to estimate article similarity. Equation (4) has two parameters, \(k_{1}\) and b. Following the suggestion of [18], they are set to 8 and 1.0, respectively, and normalization is conducted so that \(OK\) similarity is in the range of \([0,1]\). We test two versions that cluster articles based on \(OK\) similarity on titles and abstracts (\(OK\)-\(TA\)) and reference titles (\(OK\)-\(RT\)), respectively. Many previous studies have found that titles of articles are more topic-indicative than their abstracts, and hence, following [23; 24], words from titles are added twice to give them higher weights in computing inter-article similarity. To employ these state-of-the-art similarity measures to conduct overlapping clustering, _fuzzy C-medoids_ clustering (FCMdd) is employed. As noted in Section 1, FCMdd works to find representative data objects (i.e., _medoids_) for clusters, so that sum of within-cluster dissimilarities is minimized. To investigate whether ICRT can further improve well-tuned underlying clustering systems, the number of clusters produced by FCMdd is set to the number of target associations that have been identified by CTD experts (ref. Section 4.1). FCMdd is suitable for our experimental purpose, as it can conduct _overlapping_ clustering on _relational_ data (i.e., articles and similarities between them) so that state-of-the-art inter-article similarity measures (e.g., \(BC\), _DescriptiveBC_, \(PMS\), and \(OK\) noted above) can be used for article clustering, while many other clustering techniques only worked on distance measures defined on a vector space (e.g., fuzzy c means [5]). In the results of fuzzy clustering produced by FCMdd, each article has different degrees of relatedness to the clusters. Therefore, to determine the final clusters of articles, thresholding is conducted to tune a relatedness threshold in the hope to achieve the best performance in the evaluation criteria to be defined in Section 4.3 (i.e., \(BCubedF1\), see Equation (21)). As noted in Section 4.1, training data is used to tune the best thresholds for the systems. As noted in Section 2.2, another typical kind of content-based similarity measure can be derived based on topic modeling, which identifies latent topics in a collection of articles. The topic modeling systems are implemented with a toolkit that is available at [34]. To investigate whether ICRT can further improve well-tuned systems based on topic modeling, the number of topics is set to the number of target associations that have been identified by CTD experts (ref. Section 4.1). As an article can be a mixture of the topics, it can be represented as a topic proportion vector in which a dimension corresponds to the degree of relatedness to the corresponding topic (cluster). Therefore, thresholding on the degrees of relatedness is conducted to determine the final clusters as well, and as for \(OK\) noted above, there are two versions, _TM-TA_ and _TM-RT_, that respectively apply topic modeling to the two different textual contents (i.e., titles and abstracts, as well as reference titles).