## 4 Experiments
### Results  
Figures 4-6 respectively show test results on three datasets: chemical-gene, chemical-disease, and gene-disease datasets defined above (ref. Section 4.1). ICRT successfully works as a post-processor to improve all the underling systems in all the datasets. It is interesting to note that, with the help of ICRT, all the systems achieve similar performance on individual datasets, indicating that ICRT can work with various clustering systems to conduct issue-based clustering. Among the three datasets, these systems achieve worse performance on the chemical-gene dataset (compare \(BCubedF1\) values in Figures 4-6). As noted in Section 4.1, in this dataset, many articles belong to more clusters (about 50% of the articles belong to three or more clusters, ref. \"# articles belonging to X clusters\" of the first dataset in Table 1), making it more difficult to identify appropriate clusters. Overlapping clustering of the articles is thus an essential and challenging task.  
To further investigate whether the performance difference between an underlying system U and the enhance system U+ICRT is statistically significant, Equations (22)-(24) are defined to respectively estimate precision (\(P_{article}\)), recall (\(R_{article}\)), and F1 (\(F1_{article}\)) on each _individual_ article, where \(P_{pair}\) and \(R_{pair}\) have been defined above (ref. Equations (17) and (18)). Note that \(R_{article}\) is always computable, becasue all experimental articles belong to certain target clusters. However, \(P_{article}\) for an article \(a\) may be incomputable, because the system may not put \(a\) into any cluster. In this case, \(P_{article}\) is set to 1.0 [37; 38]. Moreover, when both \(P_{article}\) and \(R_{article}\) for an article are 0, \(F1_{article}\) is set to 0 as well [38], becasue in that case the clustering result for the article is totally incorrect. A paired t-test with 99% confidence level is then conducted, and the results show that all the performance differences are statistically significant,reconfirming the contribution of ICRT to the identification of article clusters corresponding to the research issues (i.e., associations between specific entities) curated by CTD experts.  
\[P_{article}(a) =Avg_{b\in A;b\
eq a;C(a)\cap C(b)\
eq\phi}P_{pair}(a,b) \tag{22}\] \[R_{article}(a) =Avg_{b\in A;b\
eq a;C(a)\cap G(b)\
eq\phi}R_{pair}(a,b)\] (23) \[F1_{article}(a) =\frac{2\times P_{article}(a)\times R_{article}(a)}{P_{article}(a )+R_{article}(a)} \tag{24}\]  
It is also interesting to note that, several clustering systems that work on titles and abstracts (i.e., those that have 'TA' in their names, including _OK-TA_, _TM-TA_, _TM-TA-FC_) perform worse than their counterparts that work on reference titles (i.e., those that have 'RT' in their names, including _OK-RT_, _TM-RT_, _TM-RT-FC_) on all the datasets. Moreover, _BC_ and _DescriptiveBC_, which work on article references, have overall better performance among the underlying systems. Therefore, article references are helpful for issue-based clustering. ICRT significantly improves these systems with a more effective method to employ titles of the references. _PMS_, which employs several well-tuned indicators to estimate article similarity on titles and abstracts (ref. similarity measure of PubMed noted in Section 2.2), can be further improved by ICRT as well, indicating that reference titles can provide helpful information that is different from the information extracted from titles and abstracts of articles.  
Figure 4: Test results on the chemical-gene dataset.  
Figure 5: Test results on the chemical-disease dataset.