# ShuffleDetect: Detecting Adversarial Images against Convolutional Neural Networks
## 9 Intrinsic Performance of ShuffleDetect\({}^{\mathbf{C},R_{th},t}\)
This is the case if after some rounds the running threshold reaches such a value that the remaining rounds, whatever happens, cannot ensure that the threshold ratio \(R_{th}\) will not be reached. The specific value chosen for \(R_{th}\) clearly impacts the different indicators of the ShuffleDetect algorithm (but, foremost, FPR and DR). To summarize, the smaller the \(R_{th}\), the higher the DR and FPR. However, our experiments show that a high \(R_{th}\) leads to a very good DR and a moderate FPR. A b-moll to this statement is that the situation differs according to the nature of the \"targeted\" or \"untargeted\" attack, as we shall see now. For targeted attacks, Table 6 together with Figure A2 led us to consider (for \(t=100\) permutations) four choices for the value for \(R_{th}\):* \(R_{th}=51\%\) matches the requirement that most permutations declare an image as adversarial. * \(R_{th}=54\%\) is motivated by the fact that the smallest \(s_{min}^{target}(k,atk)\) among the 100 permutations and the four targeted attacks is \(\geq\)54. * \(R_{th}=87\%\) is motivated by the fact that the average of the \(s_{min}^{target}(k,atk)\) among the 100 permutations and the four targeted attacks is =87. * \(R_{th}=91\%\) as a demanding ratio compromise. For untargeted attacks, Table 7 together with Figures 13 - 13 (Appendix C) show that (for \(t=100\) permutations) using \(s_{min}^{untarget}(k,atk)\) is irrelevant for the selection of \(R_{th}\). Indeed, \(s_{min}^{untarget}(k,atk)\) is usually small. More precisely, \(s_{min}^{untarget}(k,atk)\leq 35\) in all cases, and is = 9.8 on average, as opposed to what occurs for the target scenario, where \(s_{min}^{target}(k,atk)\geq 54\) in all cases, and is = 87 on average. Therefore, we limit the selection of \(R_{th}\) to two values:  
* \(R_{th}=51\%\) for the same reason as for the target case. * \(R_{th}=91\%\) because it makes sense to keep the same demanding \(R_{th}\) value for the detector independently on the scenario of the attack, hence the same value for the targeted attack. The values of FP and FPR depend only on the value of \(R_{th}\) (since no attack is considered for their computation), and on the CNN. Note that FPR = FP/100 for \(t=100\) permutations. One writes FP\({}_{avg}\) and FPR\({}_{avg}\) for their respective average values over the 10 considered CNNs. Table 8 provides the corresponding values for \(R_{th}=51\%,54\%,87\%\), and \(91\%\) (the four values used in the context of the target scenario also contain the two values used in the context of the untargeted scenario) for \(t=100\) permutations. **Remark. ** The number of adversarial images against each CNN, created either by targeted or untargeted attacks, is in all cases strictly less than the number of clean ancestor images from which these attacks started. As mentioned at the end of Section 3, this imbalance should be considered to have a fair comparison basis and sound values for the indicators (what we measure is the performance of the indicator, not of the attack). Therefore, in Tables 9-12 the clean images selected are those that correspond to the adversarial images obtained from them. For instance, since the EA-targeted attack succeeded to create \"only\" 91 images adversarial against \(\mathcal{C}_{1}\), we consider only the exact 91 clean images from which these adversarial images were obtained to assess the FP value. For targeted attacks, Tables 9 and 10 show (for \(t=100\) permutations) the DR (which coincides with \(\Psi_{\mathcal{C}_{k}}(100,100R_{th},\Omega_{atk,k}^{target})\)), TP, FN, precision, recall, and F1 score values per CNN per targeted attack for each of the four selected values of \(R_{th}\), as well as their average values. For untargeted attacks, Tables 11 and 12 show (for \(t=100\) permutations) the DR (which coincides with \(\Psi_{\mathcal{C}_{k}}(100,100R_{th},\Omega_{atk,k}^{untarget})\)), TP, FP, FN, precision, recall, F1 scores per CNN per untargeted attack for each selected value of \(R_{th}\), as well as their average values.