## 4 Experiments
### Potential Application of BCCCC to Biomedical Search Engines  
We further investigate the potential application of BCCCC to biomedical search engines by comparing its performance with PMS (a PubMed service), which is a service provided by PubMed to recommend related articles for a given article. As noted in Section 2, PubMed is a popular search engine for biomedical professionals, and PMS integrates several kinds of well-known indicators [23; 24]. These indicators are routinely employed in information retrieval systems as well. PMS was also one of the best to cluster scientific articles as well [22]. Therefore, by comparing how BCCCC and PMS identify highly related articles, the potential contribution of BCCCC to biomedical article recommendation can be evaluated. For each target article, related articles recommended by PubMed were collected on 18 October and 19 October 2019. We focused on those target articles for which the numbers of recommended articles were less than 200. Some of the articles recommended for a target article \(a_{T}\) may not be the test articles in the experiments, and hence there is no conclusive information to validate whether these articles are highly related to \(a_{T}\). Therefore, to conduct objective performance comparison, these articles  
Figure 8: Contribution of _category-_based CC when compared with traditional CC, which employs _article-_based CC (_*_’ indicates that performance difference is statistically significant with \(p<0.01\)). are excluded so that both BCCCC and PMS can work on the same set of test articles whose relatedness to \(a_{T}\) has been validated by CTD experts. More specifically, given a target article \(a_{T}\), let \(P_{T}\) be the set of articles that are recommended by PubMed and included in the test articles judged by CTD experts. Let \(l\) be the _lowest_ rank of the articles (in \(P_{T}\)) in the ranked list produced by BCCCC, and \(B_{T}\) be the set of articles that BCCCC ranks at the 1 st to the \(l\)th positions. Therefore, \(B_{T}\) includes articles in \(P_{T}\), as well as those that are ranked higher by BCCCC. Articles in \(B_{T}\) thus fall into two types: (1) those that are recommended by both BCCCC and PMS (i.e., the set \(P_{T}\)); and (2) those that are recommended by BCCCC but _not_ PMS (i.e., the difference set \(B_{T}\)-\(P_{T}\)). With this experimental setting, it is reasonable to expect that PMS actually prefers the former (i.e., articles in \(P_{T}\)) to the latter (i.e., articles in \(B_{T}\)-\(P_{T}\)). Performance of PMS and BCCCC can be compared by measuring their precision and recall on \(P_{T}\) and \(B_{T}\), respectively. Precision of PMS is the percentage of highly related articles in \(P_{T}\), while precision of BCCCC is the percentage of highly related articles in \(B_{T}\) (see Equations (13) and (15)). Computation of recall requires the number of highly related articles that should be retrieved. This number can be seen as the highly related articles in \(B_{T}\) (see denominators of Equations (14) and (16), and if there is no highly related articles in \(B_{T}\), the target article \(a_{T}\) is excluded from the experiment), because BCCCC recommends all articles in \(B_{T}\), while PMS may only recommend some of them (as noted above). Recall of BCCCC is thus always 1.0 (see Equation (16)), and we are investigating whether this is at the cost of recommending more articles and thus possibly reducing its precision.