# Evading Logits-Based Detections to Audio Adversarial Examples by Logits-Traction Attack
## 4 Experimental Evaluation
### Logits Dispersion Detection Effectiveness Evaluation  
When the detection algorithm is deployed, the defender only has information about the model to be protected, and it does not know the data source of the original speech used by the attacker. Therefore, the hyperparameter \(p\) and threshold \(k\) set by each detection algorithm should be fixed, and only change with the protected model. The current test parameters are set as follows:  
1. Logit-\(\Delta_{12}\) detection. It does not require a hyperparameter \(p\). If the average difference between the largest and second largest values exceeds the threshold \(k\), it is determined that the logits belong to the original speech;
2. Logit-\(\sigma^{+2}\) detection. In DeepSpeech, \(p\) is set to 0, which means to calculate the variance of the numbers whose logit value is greater than 0 in all decision frames. If \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Attack** & **Performance** & **Libri** & **MCV** \\ \hline \multirow{3}{*}{C\&W} & Accuracy & 81.6 & 66.1 \\  & FPR & 2.6 & 48.6 \\  & FNR & 37.6 & 39.2 \\ \hline \multirow{3}{*}{LT **(Ours)**} & Accuracy & 52.1 & 28.5 \\  & FPR & 2.6 & 48.6 \\ \cline{1-1}  & FNR & 93.7 & 94.4 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Comparison of detection accuracy (%) on the LOGITS NOISE method. the variance is greater than the threshold \(k\), it is determined that the logits belong to the original speech. Similarly, \(p\) is set to 10 in Lingvo;
3. Logit-\(c^{+}\) detection. In DeepSpeech, \(p\) is set to 9, which means to count the number of values greater than 9 in all decision frames. If the number is greater than the threshold \(k\), it is determined that the logits belong to the original speech. Similarly, \(p\) is set to 14 in Lingvo. The detection results of the three detection methods on the C&W adversarial examples are presented in Table 2:  
It can be seen from Table 2 that all three features have high accuracy on the LibriSpeech dataset, and the three features in the DeepSpeech and Lingvo models have similar detection accuracy distributions, indicating that our three detection methods based on logit dispersion are efficient. We take the detection using \(\sigma^{+2}\) features on the Lingvo system as an example, which achieved the highest detection accuracy on the LibriSpeech dataset. Their detection results are shown in Figure 7:  
The \(\sigma^{+2}\) of C&W adversarial examples less than 12 can be correctly detected. A few normal speech values less than 12 are mistakenly judged as adversarial examples. There is a clear dividing line between this two. The threshold \(k=12\), is the \(\sigma^{+2}\) dividing line between the adversarial example and normal speech. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Attack** & **Performance** & \multicolumn{2}{c}{**DeepSpeech**} & \multicolumn{2}{c}{**Lingvo**} \\  & & **Libri** & **MCV** & **Libri** & **MCV** \\ \hline \multirow{3}{*}{Logit \(-\Delta_{12}\)} & Threshold & \multicolumn{2}{c}{\(k=6\)} & \multicolumn{2}{c}{\(k=6\)} \\  & Accuracy & 89.7 & 61.9 & 97.4 & 78.1 \\  & FPR & 8.4 & 47.4 & 5.2 & 41 \\  & FNR & 12.1 & 28.4 & 0 & 1.8 \\ \hline \multirow{3}{*}{Logit \(-\sigma^{+2}\)} & Threshold & \multicolumn{2}{c}{\(k=12\)} & \multicolumn{2}{c}{\(k=12\)} \\  & Accuracy & 91.1 & 58.5 & 96.7 & 75.8 \\ \cline{1-1}  & FPR & 10.4 & 63.6 & 2.6 & 35.6 \\ \cline{1-1}  & FNR & 7.5 & 18.3 & 4.0 & 12.3 \\ \hline \multirow{3}{*}{Logit \(-c^{+}\)} & Threshold & \multicolumn{2}{c}{\(k=11\)} & \multicolumn{2}{c}{\(k=8\)} \\  & Accuracy & 89.7 & 62.9 & 88.3 & 82.5 \\ \cline{1-1}  & FPR & 6.0 & 46.4 & 7.4 & 23.6 \\ \cline{1-1}  & FNR & 14.5 & 27.3 & 16.0 & 11.1 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Performance (%) of our three features. Figure 7: Detecting C&W adversarial examples by \(\sigma^{+2}\) feature in Lingvo. Second, this paper uses the MCV dataset as comparison experiments, and its FNR is lower, meaning that our detection features can effectively detect even adversarial examples generated based on chaotic speech.