## 4 Experiments
### Baseline Systems for Performance Comparison with BCCCC  
Table 1 lists the baseline systems in the experiments. The first three baselines are BC and its variants: IBS (Issue-Based Similarity [9]) and DBC (DescriptiveBC [10]). They are the main baselines in the experiments, as BCCCC aims at being an improved version of BC. Instead of relying on full texts of articles that may not be publicly available [11; 12], IBS and DBC improve BC by references in the articles. Therefore, performance comparison with BC and the two variants can identify the contribution of BCCCC to further improvement of BC, which has been a fundamental technique for literature retrieval, analysis, and mapping.  
To estimate the similarity between two articles \(a_{1}\) and \(a_{2}\), the three baselines consider the references cited by \(a_{1}\) and \(a_{2}\). The key difference is that BC treats each reference in an \"object-based\" manner, because the similarity between \(a_{1}\) and \(a_{2}\) is increased only when they cocite the same objects (i.e., references, ref. Equation (1)). On the other hand, IBS and DBC were developed to improve BC by \"title-based\" similarity estimation, in which the similarity between \(a_{1}\) and \(a_{2}\) can be increased if they cite references with _similar_ titles, even though these references are _different_ from each other. IBS estimates the similarity between two articles based on a certain number of _most-similar_ references titles in the articles, while DBC estimates the similarity based on _all_ references' titles in the articles. It was shown that, by considering the title-based similarity, IBS and DBC performed significantly better than BC in article clustering [9] and article ranking [10]. Therefore, BC and the two state-of-the-art variants (IBS and DBC) can be the main baselines to verify whether BCCCC is a further improved version of BC. For more detailed definitions for IBS and DBC, the readers are referred to [9] and [10], respectively.  
Moreover, to evaluate BCCCC more comprehensively, BM25 ref is implemented as a baseline as well. This baseline represents a way that ranks articles by text-based similarity on reference titles. As noted in Section 2, BM25 [21] was one of the best text-based techniques to identify related scientific articles [22]. We apply BM25 to estimating the similarity between _concatenated reference titles_ (CRTs) of articles. For each article, a CRT is constructed by concatenating all titles of the references cited by the article. The similarity between a target article \(a_{T}\) and another article \(a_{x}\) is simply the BM25 similarity between their CRTs (denoted by \(\text{CRT}_{T}\) and \(\text{CRT}_{x}\), respectively). BM25 ref similarity is defined in Equation (7), where \(k_{1}\) and \(b\) are two parameters, \(|CRT|\) is the number of terms in \(CRT\) (i.e., length of \(CRT\)), _avalgen_ is the average length of CRTs (following several previous studies [10; 22], the two parameters \(k_{1}\) and \(b\) of BM25 ref are set to 2 and 0.75 respectively).  
\[\text{BM25 ref}(a_{T},a_{x})=\sum_{t\in CRT_{T}\cap CRT_{x}}\frac{TF(t,CRT_{x })(k_{1}+1)}{TF(t,CRT_{x})+k_{1}(1-b+b\frac{\text{\small{\small{\small{\small{ \small{\small{\small{\small{\small{\small{\small{\small{\small{\small{\small{\small{ \small{{\small{{    { {    }}}}} {{}}_{{{}}_{{}}_{{}_{{}}}}}}}}}}}}}})})Log_{2}IDF(t) \tag{7}\]  
Therefore, BM25 ref is _not_ a version of BC, although it relies on the references cited by articles as well. It is thus actually _not_ the main baseline in the experiments. However, comparison of BCCCC and BM25 ref can provide additional evidence to further validate whether BCCCC can perform better than a typical text-based approach, which works on reference titles as well. BCCCC can be an enhanced version of BC only if it performs significantly better than all the baselines.