## 4 Experiments
### Evaluation Criteria  
As noted above, each article will be a target article exactly one time. Therefore, for each target article, we evaluate how the systems rank its highly related articles (i.e., those that are judged by CTD experts to be focusing on the same research topic as the target article). Two evaluate criteria that are commonly employed by previous studies (e.g., [10]) are employed to evaluate the systems. The first criterion is Mean Average Precision (MAP), which measures how highly related articles are ranked at higher positions. MAP is defined in Equation (8), where \(T\) is the set of target articles, and AvgPrecision(\(i\)) is the average precision for the \(i\)th target article. MAP is thus the average of the AvgPrecision values for all the target articles.  
\begin{table}
\begin{tabular}{l c c} \hline \hline \multirow{2}{*}{**Baseline System**} & \multicolumn{2}{c}{**Usage of the References**} \\ \cline{2-3}  & **Individual** & **Concatenated** \\ \cline{2-3}  & **Object-Based** & **Title-Based** & \\ \hline (1) BC (Bibliographic Coupling) & \(\surd\) & \\ (2) IBS (Issue-Based Similarity) & & \(\surd\) \\ (3) DBC (DescriptiveBC) & & \(\surd\) \\ (4) BM25 ref (BM25 on references) & & \(\surd\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Baseline systems for performance comparison with BCCCC.  
\[MAP = \frac{\sum\limits_{i=1}^{|T|}\text{AvgPrecision}(i)}{|T|} \tag{8}\]  
For each target article, AvgPrecision is defined in Equation (9), where \(H_{i}\) is the number of articles that are highly related to the \(i\)th target article, and \(Rank_{i,j}\) is the rank of the \(j\)th highly related article of the \(i\)th target article. As the system being evaluated aims at ranking articles, \(Rank_{i,j}\) is determined by the system, and hence \(Rank_{i,j}\) is actually the number of articles that readers have read when the \(j\)th highly related article is recommended by the system. The ratio \(jRank_{i,j}\) can thus be seen as the precision (achieved by the system) when the \(j\)th highly related article is shown. AvgPrecision(\(i\)) is simply the average of the precision values on all highly related articles of the \(i\)th target article. It is in the range [0-1], and it will be 1.0 when all the highly related articles are ranked at top-\(H_{i}\) positions.  
\[\text{AvgPrecision}(i)=\frac{\sum\limits_{j=1}^{H_{i}}\frac{j}{Rank_{i,j}}}{H _{i}} \tag{9}\]  
Therefore, MAP is concerned with how all highly related articles are ranked at higher positions. In some practical cases, readers may only care about how highly related articles are ranked at _top_ positions (e.g., readers only read a certain number of articles at top positions). Therefore, another evaluation criterion average P@X is employed as well. This criterion considers those articles that are ranked at top-X positions only. It is defined in Equation (10), where P@X(\(i\)) is the precision when top-X articles are shown to the readers for the \(i\)th target article (as defined in Equation (11)). As readers often care about a limited number of top positions only, X should be set to a small value, and hence we investigate performance of the systems when X is set to 1, 3, 5, and 10.  
\[\text{Average P@X}=\ \frac{\sum\limits_{i=1}^{|T|}\text{P@X}(i)}{|T|} \tag{10}\]  
\[\text{P@X}(i)=\ \frac{\text{Number of top}-\text{X}\text{articles that are highly related to the $i^{th}$ target article}}{\text{X}} \tag{11}\]  
By simultaneously measuring performance of the systems in both MAP and average P@X, we can comprehensively evaluate how the systems rank all highly related articles, as well as how highly related articles are ranked at top positions. A better system should be able to perform significantly better than others in both evaluation criteria.