# A Deep Learning Method for Lightweight and Cross-Device IoT Botnet Detection +
## 5 Results
### Discussion of the Results  
In the following, we complement the results presented above by a visual analysis and a discussion on the operation of the two considered approaches.  
Figure 3 shows the reconstruction error (RE) obtained on the test set of the Danmini doorbell and the Provision PT-838 security camera devices by considering the separate autoencoding approach (Figure 3a,c) and the all-in-one autoencoding approach (Figure 3b,d). We limit the visual analysis to these two devices because all the remaining cases lead to similar findings. The REs of normal and attack points of the test sets refer to the separate and all-in-one experiment, respectively. Each data point is marked by either \(\bullet\)--normal data point-- or \(\blacktriangledown\)--attack data point--for better visualization; the x-axis is the id of the point in the test set; the y-axis is the corresponding RE. A semi-logarithmic scale (x-axis in linear scale and y-axis in log scale) is used to better visualize the REs. The horizontal dashed line shows the anomaly threshold (a by-product of the training phase).  
In general, according to Figure 3, it can be noted that both autoencoding approaches--separate and all-in-one--return a low RE for almost all the normal points, which are thus below the detection threshold; on the other hand, most of the attack points lead to high RE and are well above the threshold. Overall, Figure 3 is useful to gain a visual understanding of true negatives (NORMAL points below the threshold), false positives (NORMAL points above the threshold), false negatives (ATTACK points below the threshold) and true positives (ATTACK points above the threshold) of both autoencoding approaches.  
It is easy to link the visual results and the AE performance figures by analyzing the recall and FPR values--Dannini and Provision PT-838 devices--reported in Table 6 (separate autoencoding) and Table 7 (all-in-one autoencoding). In particular, for the Danmini doorbell device, Figure 3a (separate autoencoding) shows a higher number of NORMAL points above the threshold--false positives--if compared with Figure 3b (all-in-one autoencoding); in fact, the FPR values for this device are 0.0108 (separate autoencoding) and 0.0004 (all-in-one autoencoding). On the other hand, Figure 3a (separate autoencoding) shows no attack points under the threshold--false negatives--if compared with Figure 3b (all-in-one autoencoding). Again, this visual finding is confirmed by the performance figures. As a matter of fact, the recall values for this device are 1.0 (separate autoencoding) and 0.9997 (all-in-one autoencoding). The same considerations can be extended to the Provision  
Figure 3: RE of the test sets for the Danmini doorbell and Provision PT-838 security camera devices: (a) Danmini–Separate autoencoding; (b) Danmini–All-in-one autoencoding; (c) Provision PT-838–Separate autoencoding; (d) Provision PT-838–All-in-one autoencoding.  
PT-838 security camera device (Figure 3c,d). It is worth pointing out that this trend--lower FPR at the expense of lower recall for the all-in-one approach--is preserved on almost all devices (as shown in Tables 6 and 7).  
Overall, we believe that the all-in-one approach has remarkable strengths. Although the recall values obtained with this approach are slightly lower than the separate approach ones, they are surely acceptable (in the range 0.9994 to 0.9997). Moreover, the FPR is always close--or even equal--to 0; on the other hand, the FPR values obtained with the separate autoencoding are worse, and they range from 0.0002 to 0.0417. In real-life environments, false positives can be cumbersome to deal with for network administrators. A high number of false positives leads to lost confidence in the alerts and to lower defense levels. Therefore, in addition to the advantages related to the scalability and the portability (a single network topology for all the devices), the all-in-one approach might also be beneficial to mitigate the false positive problem.