## 4 Experiments
### Ranking-Based Recommendation  
For this scenario, experiments are evaluated on two practical applications: music and book recommendations. The dataset adopted is from the work in [26], which includes Douban book and NetBase music datasets. Douban ([[http://www.douban.com](http://www.douban.com)]([http://www.douban.com](http://www.douban.com))) is the largest book recommendation website in China. Netease music is a freemium music streaming service developed and owned by NetEase, Inc. The distribution of the training/testing sets are also set to 90% and 10%. Table 2 provides statistics regarding these two datasets.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Algorithm** & **RMSE** & **Time (s)** & \(p\)**-Value** \\ \hline ACF & 1.853 \(\pm\) 0.02 & 1340.3 & 0.3316 \\ IACF & 1.012 \(\pm\) 0.02 & 137.2 & \\ \hline \hline \end{tabular}
\end{table}
Table 1: Rating-based recommendation results evaluation.  
\begin{table}
\begin{tabular}{c c c} \hline \hline  & **Douban** & **NetEase** \\ \hline \#User & 4965 & 115,994 \\ \#Item & 41,785 & 19,980 \\ \#Preference & 958,425 & 2,399,639 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Dataset properties.  
We evaluated the methods in terms of precision, recall, and F1. The definitions are given below.  
\[Precision=\frac{\#\;Hitting}{N},Recall=\frac{\#\;Hitting}{\#\;Likes},F1=2\times \frac{Precision\times Recall}{Precision+Recall}\]  
Precision is a widely used metric in recommender systems. Precision is the percentage of relevant items out of those items selected by the query. The value of precision is \(\in[0,1]\), with a higher value indicating a more accurate prediction [27]. Recall is the percentage of relevant items that are actually retrieved over the total amount of relevant items. F1 is the harmonic mean of precision and recall taking both metrics into account. Several ranking-based CF methods, NBI [28], RSM [29], the BM25-Item algorithm [30], and the classic user-based and item-based CF algorithms are chosen for comparison. Similarly, the distribution of the dataset is 90% for training and 10% for testing. All the results were obtained by averaging five different runs. Table 3 shows the comparison. On the Douban dataset, both the ACF and IACF have better F1 than other methods. The IACF demonstrates better performance than the ACF in terms of precision and running time. On the NetEase dataset, though both the IACF and ACF are not the best in terms of F1, their F1 scores are only a bit lower than the F1 scores of other methods. Again IACF shows better performance over ACF in regard of precision and running time.  
Similarly, the statistical difference is examined regarding the IACF and ACF algorithms. The alternative hypothesis is that IACF does not improve the performance in terms of RMSE. A value of 0.05 is chosen for alpha, which means the results are significant if the \(p\)-value is below 0.05. The standard deviations observed for the experiments that are run on Douban and NetEase datasets are 3.1 and 4.3, respectively. With these measurements and the precision in Table 3, the \(p\)-values calculated are 0.4962 and 0.4989. There is statistically significant evidence showing that the IACF is an improvement over the ACF.