# Defending against FakeBob Adversarial Attacks in Speaker Verification Systems with Noise-Adding +
## 6 Performance Evaluation
### Performance Evaluation of the Proposed Defense System against FakeBob Attacks  
Next, we study the performance of our proposed defense methods against FakeBob attacks presented in Algorithm 1 in an SV system. Specifically, we use the ASR as the performance metric. A smaller value of the ASR indicates a better defense performance. Moreover, FakeBob uses multiple iterations to create an adversarial audio, as shown in Algorithm 1. Therefore, we also measure the average number of iterations and the average running time for FakeBob attacks to find an adversarial audio in our experiments. A larger number of the average iterations and a longer average running time reflect a better defense system. For adversarial attacks, we randomly selected five clean audio clips from each of four illegal speakers. These 20 audios repeatedly went through an SV system for 100 times, in order to obtain the average ASR, the average number of iterations, and the average running time of FakeBob attacks. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Plugin** & \(\sigma\) & **EER (\%)** & **Time (s)** \\ \hline Original & 0 & 1.05 & 18.44 \\ \hline Denoising & 0.001 & 1.61 & 30.67 \\ \hline Denoising & 0.002 & 2.95 & 30.41 \\ \hline Denoising & 0.005 & 3.36 & 30.79 \\ \hline Noise-Adding & 0.001 & 1.21 & 19.34 \\ \hline Noise-Adding & 0.002 & 1.92 & 19.78 \\ \hline Noise-Adding & 0.005 & 3.94 & 20.31 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Performance evaluation of proposed defense system on normal operations of GMM SV systems. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Plugin** & \(\sigma\) & **EER (\%)** & **Time (s)** \\ \hline Original & 0 & 0 & 433.45 \\ \hline Denoising & 0.001 & 0.15 & 447.37 \\ \hline Denoising & 0.002 & 0.05 & 447.82 \\ \hline Denoising & 0.005 & 0.49 & 446.20 \\ \hline Noise-Adding & 0.001 & 0.44 & 435.35 \\ \hline Noise-Adding & 0.002 & 0.39 & 435.89 \\ \hline Noise-Adding & 0.005 & 1.14 & 435.51 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Performance evaluation of proposed defense system on normal operations of I-Vector SV systems. Table 3 shows the experimental results of our proposed defense functions, i.e., denoising and noise-adding with Gaussian noise, against FakeBob attacks in a GMM SV system for speakers 2830 and 61, when \(\sigma=0\), 0.001, 0.002, and 0.005. It can be observed that while FakeBob achieves 100% ASR for the original GMM SV system (i.e., \(\sigma=0\)), the ASR of the SV system with the defense is less than 100%. Moreover, when \(\sigma\) increases, the ASR decreases in general. For example, when \(\sigma=0.002\), the average ASR is 56.05% for the denoising function, whereas it is only 5.20% for the noise-adding method. It can be clearly observed that noise-adding performs much better than denoising based on the ASR. Moreover, noise-adding leads to a much larger value of the average number of iterations and much longer average running time than denoising and the original SV system. The average number of iterations for denoising is similar to that in the original SV system, whereas the average running time for denoising is slightly larger than that in the original SV system. However, with the noise-adding plugin, the values of these two metrics are much larger. For example, in the original GMM SV system, the average number of iterations is 23.00, and the average running time is 158.68. However, in the defense system with noise-adding and \(\sigma=0.002\), the average number of iterations and the average running time are 604.95 and 3992.88, respectively. This indicates that noise-adding slows down the attacker's processing speed more than 25 times and makes FakeBob significantly harder to find the adversarial audios. Moreover, we have extended our experiments to an i-vector SV system against FakeBob attacks. As shown in Table 4, denoising and noise-adding with \(\sigma=0.002\) are able to reduce the ASR from 95.00% to 38.63% and 0.50%, respectively.