# Anomaly Detection of CAN Bus Messages Using a Deep Neural Network for Autonomous Vehicles
## 4 Experimental Results
### Performance Evaluation  
At the same time, to demonstrate the better performance of proposed method, we compare the proposed method with two other methods: DNN + support vector machine (SVM) and DNN + Softmax. The former uses a deep network to train the original data, and then classifies it by embedding SVM as the back-end. The basic principle of SVM is to find the best separating hyperplane in feature space to maximize the interval between positive and negative samples in a training set. SVM is a supervised learning algorithm to solve binary classification problems. After introducing the kernel method, SVM can also be used to solve non-linear problems. The latter also uses a deep network to train the original  
Figure 5: The number of a special bit-symbol in the data field of 8 bytes. data, and then classifies the data by embedding the Softmax function as the back-end. Softmax is a very common and important function in deep learning, especially in the field of classification. It maps some inputs to real numbers between zero and one, and guarantees the normalization sum to one, so the sum of probability of classification is just one. The anomaly detection stage is completed at the back-end. In this stage, the similarity between any two sets of CAN bus sequences and the calibrated sequence is compared to determine which group of sequences is normal and which group of sequences is invaded. In our experiment, all comparative experiments are performed under the same hardware and software conditions. To evaluate the performance of the proposed method, the front-end of the three methods proposed in this paper all adopt the same deep network structure; that is, they all have the same number of layers and neurons. In addition, the training parameters are shared with each other. Among the three methods, the learning rate of the network training stage is 0.001. Figure 6 shows the performance changes of the three methods mentioned in this paper as the number of hidden layers increases. It can be seen that with the increase in the number of layers, the recognition accuracy of the three methods is improved, but when the number of layers increases to a certain level, the improvement gradually tapered off. In terms of performance, under the same number of hidden layers, the method that uses the triplet loss function as the back-end is the best, followed by the method using SVM as the back-end. The worst performance was from the method that used Softmax as the back-end. In order to compare the performance of the three methods more intuitively, Figure 7 shows a box chart that provides a statistical graph of data distribution, in which the three different color boxes represent the performance of the three methods. It can also be seen that the performance rankings of the three methods are DNN + Triplet, DNN + SVM and DNN + Softmax. Specifically speaking, breaking through the limitation of hidden layer number, the lower limit and upper limit of detection accuracy using DNN + Triplet architecture are higher than the other two architectures. In addition, from the median detection accuracy of the three methods, the architecture of DNN + Triplet is also significantly higher than the other two architectures. Figure 6: Performance changes with the increase of neural network layers. DNN = deep neural network; SVM = support vector machine. We also show the time cost to perform the detection task, which depends on the different number of hidden layers as shown in Figure 8. It can be seen that the time cost increases with the increasing number of hidden layers, and the whole detection phase is about 2-19 s, in which the training phase takes up almost the entire phase of time consumption. Training time represents the measurement time, which is required to train the DNN structure in the training phase. The time cost of feature extraction represents the process consumption of transforming data features into feature vectors. The time cost of detection represents the process consumption of similarity calculation in terms of the triplet loss network.