# Boosting the Learning for Ranking Patterns
## 6 Experiments
### Active Learning Results  
In this section, we aim to address research questions **RQ2** and **RQ3**, which involve comparing the active learning versions of RankingSVM and AHPRank. To achieve this, we  
Figure 3: CPU time comparison between RankingSVM.0 and AHPRank.0 learning using the Chi-Emu emulator on different training data sizes averaged over all the datasets. Figure 2: Learning accuracy comparison between RankingSVM.0 and AHPRank.0 learning Chi-Emu on different training data sizes. employ a straightforward process that involves asking the user ranking queries on pairs of patterns (e.g., do you prefer \(P_{i}\) to \(P_{j}\)?). **(A) Evaluating the effectiveness of our sensitivity-based heuristic. ** In this section, we aim to evaluate the effectiveness of our sensitivity-based generator (SBG) (see Section 4.5) in improving the active learning process of AHPRank.1. To do so, we compare SBG to a random generator (RG), where RG randomly selects a pair of patterns from #Rules and submits them to both RankingSVM.1 and AHPRank.1. We conduct our experiments with human-in-the-loop, setting the number of queries (i.e., iterations of Algorithm 2) to a maximum of 20 queries. We repeat the experiment 10 times and take the average result to account for the sampling step of our SBG and the randomness of the RG approach. After a few tests, we set the sample size \(\mathcal{X}\) picked in line 10 of Algorithm 2 to \(\theta=10^{3}\), providing a good trade-off between time selection and the accuracy of the selected pair. Figure 4 presents a scatter plot of 20 iterations comparing the performance of RankingSVM.1 and AHPRank.1 with both RG and SBG. We report the Spearman correlation \(\rho\), recall at 10%, and recall at 1% for each iteration. The results show that the use of SBG outperforms RG, regardless of the learning algorithm, as can be observed from the scatter plot. To support our observation on the effectiveness of SBG, we conducted a statistical test using the Wilcoxon signed-rank test. We used a one-tailed alternative hypothesis with the null hypothesis that RG is more efficient than SBG: H0: The accuracy using RG \(\geq\) the accuracy using SBG. The alternative hypothesis H1 states that SBG outperforms RG. With this statistical test, we concluded that the use of SBG is more efficient than RG (i.e., H1 is accepted). Table 6 reports the \(p\) value, z-score, and confidence interval (CI) of each test. Except for the case of (RankingSVM.1, R@10%), we had strong evidence to reject the null hypothesis based on the \(p\) value column. The CI column clearly shows that the use of SBG is better than RG. Consequently, we will use the SBG heuristic in RankingSVM.1 and AHPRank.1 in the following. Figure 4: Qualitative comparison of the random vs. sensitivity-based query generator using the Spearman correlation \(\rho\) and the recall at 10% and 1% of each iteration. The main observation from Table 7 is that AHPRank.1 outperforms RankingSVM.1. Looking at the Hepatitis dataset, for example, AHPRank.1 achieves an accuracy of 78% with only 10 queries, while RankingSVM.1 achieves 73%. At 50 queries, AHPRank.1 achieves an accuracy of 95%, compared to 94% for RankingSVM.1. At 100 queries, AHPRank.1 reaches 98% accuracy, while RankingSVM.1 remains stable at 94%. In terms of recall, AHPRank.1 discovers the most relevant patterns in the first 50K and 5K patterns (out of 500K) with accuracies of 78% and 66% over 10 queries, respectively, compared to 61% and 44% for RankingSVM.1.