# Automotive Cybersecurity Application Based on CARDIAN
## 5 Experimental Results  
In this section, the testing phase will be described and the results obtained will be highlighted. In the previous section, it was shown how the entire system was built in order to evaluate the performance of a Bayesian network in recognizing intrusions starting from a sequence of CAN messages present on the bus. These messages, in the situation examined, contain all the information relating to the status of the vehicle. The calculation of the prior and conditional probabilities of each node was carried out in Python, using the Pomegranate library. The Pomegranate module deals with implementing probabilistic models from the simplest, such as probability distributions, to the most complex, such as Bayesian networks. This module was developed on the idea that all probabilistic models can be represented as probability distributions. This vision allows the various models to be developed in a light and flexible way. Furthermore, each model has numerous integrated features, including various types of learning. It allows us to implement Bayesian networks that use different types of distributions on each feature. Finally, with a few commands, starting from a known number of pieces of evidence, it is able to determine the posterior probability associated with all the nodes of the network. The probability distributions were determined from the acquired and pre-processed data. The training dataset was constructed as described in [24]. Four types of attacks have been defined:  
* DoS attack: messages with CAN ID composed of only zeros and a random data field are injected every 0.3 milliseconds;
* Fuzzy attack: messages with CAN ID composed of random ID and random data field are injected every 0.5 milliseconds;
* Gear attack: the message with ID and SPN relating to the current gear is injected, with a random data field every millisecond;
* RPM attack: the message with ID and SPN relating to the current RPMs is injected, with a random data field every millisecond. For each interval containing one or more malicious messages, the attack value is set to 1; otherwise, it is set to 0. The number of messages acquired before data entry is 10,900,724; from these data, another four datasets were generated, one for each type of attack. Each data set is composed of the first 50% (5,450,362 messages in 150 h) of normal operation; the remaining part presents the injection of attacks. The network was then trained with a total of 20,058,781 message frames, of which 19,781,678 were malicious. Once the network was trained, it was tested first by using a real dataset lasting 1 h, and then with datasets containing the previously described attacks, of the same duration. The tests were carried out by sending messages directly to the CAN bus of the device using a USB-CAN converter. The first important parameter to consider is the system response time. The timestamps of the instant in which the last message of the time window is sent and the timestamp in which the processing produces the json file were analyzed. To classify the results obtained, the following cases were distinguished:  
* True Positives (TP): presence of an attack and correct reporting of this;
* True Negatives (TN): absence of attacks and recognition of this absence;
* False Positives (FP): absence of attacks but reports of attacks;
* False Negatives (FN): presence of an attack but no recognition of it. Using the confusion matrix, the reference parameters such as precision, recall and f1-score were calculated. The first case analyzed, a dataset without malicious messages, produced the following results: (Tables 4-12) \begin{table}
\begin{tabular}{c c c} \hline \hline
**Attacks/Detected** & _Yes_ & _Not_ \\ \hline _Yes_ & 27,311 & 132 \\ \hline _No_ & 42 & 1736 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Fuzzy attack. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Attacks/Detected** & _Yes_ & _Not_ \\ \hline _Yes_ & 50,117 & 110 \\ \hline _No_ & 35 & 1740 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: RPM attack.