## 4 Experiments
### Performance Evaluation
#### 4.6.2 Number of Votes & Substitution Rate  
The number of votes \(k\) and the substitution rate \(p\) are hyperparameters that can be tuned to enhance the performance of our detector. Figure 5 shows a comparison of accuracy between original and adversarial examples on the Word-CNN classifier trained with the IMDB dataset. Subfigure (a) presents a comparison of accuracy across different substitution rates, while subfigure (b) examines the effect of varying the number of votes. We first compare the accuracy of original and adversarial examples separately by changing the number of votes and substitution rates. According to Figure 5, when the substitution rate is low (\(p<10\%\)), it shows high accuracy in original examples and low accuracy in detecting adversarial examples. This is expected behavior, as low substitution rates will fail to detect adversarial perturbations. This is because substituted words will not always get the highest IGs attribution scores. Hence, increasing the number of substituted token counts will increase detection accuracy. When \(p\) was higher than 10, we can observe that the accuracy of the original example dropped but remained over 90%. This indicates that our method maintains good accuracy on original examples even with a higher \(p\). On the other hand, when \(p\) increases from 1% to 5%, there is a significant increase in adversarial detection accuracy. Afterwards, it remains consistent over values of 70% even with the increase in \(p\)  
Figure 4: Comparison of F1 scores on adversarial example detection against baseline methods and ours using BAE attack. (**a**) AG News, (**b**) SST-2, (**c**) IMDB.  
Figure 3: Comparison of F1 scores on adversarial example detection against baseline methods and ours using TextFooler attack. (**a**) AG News, (**b**) SST-2, (**c**) IMDB.  
Figure 2: Comparison of F1 scores on adversarial example detection against baseline methods and ours using PWWS attack. (**a**) AG News, (**b**) SST-2, (**c**) IMDB.  
According to Figure 5, there is no significant impact on the number of votes against the accuracy. We can observe a slight accuracy drop when the number of votes increases.  
Next, we analyzed the effect of \(k\) (number of votes) and \(p\) (substitution rate) on the final detection accuracy. Figure 6 presents this comparison, showing detector accuracy across varying numbers of votes and substitution rates on the Word-CNN classifier trained with the IMDB dataset. It evaluates the performance of our proposed method (subfigure (a)) against the baseline RS&V method (subfigure (b)). According to Figure 5a, we can observe a significant increase in the overall accuracy when the substitution rate is increased from 1% to 5%. This is due to the increase in accuracy in adversarial examples, which we can observe in Figure 5. Upon examining the influence of the vote count, we observed a slight decrease in accuracy as the number of votes increased. Hence, tuning these parameters is required to achieve better performance. As shown in Figure 6, having a \(p\) value in the range of 10% to 30% and the number of votes in the range of 1 to 10 demonstrates ideal performance.  
Next, we compared the effectiveness of our approach against the RS&V method by varying the \(p\) and \(k\). As shown in Figure 6b, the RS&V method performed better when \(p\) and \(k\) increased. To achieve par performance with our approach, the RS&V method requires the \(p\) to be over 60%, while our method only requires less than 10%. This is a considerable performance gain in our method as substituting large amounts of tokens with a high number of votes is a computationally expensive task that could result in overall performance degradation while taking more time to detect adversarial examples. Higher accuracy with a lower substitution rate and vote number is another advantage over the other methods.