# Defending against FakeBob Adversarial Attacks in Speaker Verification Systems with Noise-Adding +
## 6 Performance Evaluation
### Experimental Setup  
The experiments were run through three virtual machines (VMs) provided by Google Cloud Platform [43] and a local GPU server. Two of three VMs are with 8-core Intel Xeon CPU 3.1 GHz and 32 GB memory (i.e., c2-standard-8 machine type), whereas the third VM is with 16-core Intel Xeon CPU 3.1 GHz and 64 GB memory (i.e., c2-standard-16 machine type). The GPU server is with 24-core Intel I9 CPU 2.9 GHz, 128 GB memory, and GeForce RTX 2080 Ti graphic card. All machines are installed with Ubuntu 20.04.  
Our experiments focus on GMM and i-vector SV systems. These SV systems are implemented by the Kaldi speech recognition toolkit [44] and use the pre-trained models from VoxCelab 1 [45]. Moreover, the adversarial attacks against these SV systems are implemented through FakeBob attacks [7] with a perturbation threshold of 0.002 (i.e., \(\epsilon=0.002\)) and 1000 maximum iterations (i.e., \(m=1000\)). These same parameters were used in [7]. The audio dataset used is from LibriSpeech [46] and contains nine different speakers, which were also applied in FakeBob [7]. These nine speakers have five legitimate users and four illegal users, all speaking English. Legitimate users include three females and two males, whereas illegal users contain two females and two males. All users are native speakers and speak different messages. There are totally 100 audio clips from all illegal users, whereas there are about 100 audio clips from each legitimate user. These illegal audios and legitimate audios can be used to calculate the EER of an SV system. Each audio clip lasts between 4 and 10 s. The experiments on SV systems are through the application programming interface (API) provided by Kaldi GMM or i-vector.  
In [7], all legitimate users were fed into the same SV system in experiments. Different from this setup, we put every legitimate user in a stand-alone SV system and thus obtained multiple SV systems in our experiments. The final result is the average over all results from these multiple SV systems. Consequently, our obtained ASR of the FakeBob attack against an SV system without any defense can be lower than that presented in [7]. However, from our view, such a setup with multiple stand-alone SV systems is more realistic.  
It is noted that an SV system, either GMM or i-vector, without the proposed defense mechanism, is deterministic. That is, when an input audio is accepted (or rejected) by the SV system, it will be always accepted (or rejected) for future tests in the same SV system. However, an SV system that is installed with our proposed defense mechanism becomes stochastic, because of the randomness introduced by the noise generator. That is, if an input audio is accepted (or rejected) in the current test, it may be rejected (or accepted) next time when it is fed into the same SV system. To address such a stochastic effect, during the test we let an input audio go through an SV system multiple times (e.g., 10 or 100 times) and calculate the average of the performance metrics such as EER and ASR. Specifically, in Sections 6.2 and 6.3, 100 times are applied; whereas in Sections 6.4 and 6.5, 10 times are used.