# Aircraft Mode S Transponder Fingerprinting for Intrusion Detection
## 4 Performance Evaluation with Real Data  
An evaluation of the performance of the proposed approach using real data was done and described hereafter. The evaluation was based on a measurements campaign that was done between day 337 and day 345 of 2018, collecting about 45-million messages from 2942 aircraft. A simple receiver composed of a Software Defined Receiver running over a Raspberry Pi equipped with a modified Digital Video Broadcasting-Terrestrial (DVB-T) dongle and an omnidirectional ADS-B antenna [32; 33; 39] was used. The HW allowed the reception of the ADS-B messages and the storing of the relative signal samples (I and Q samples, 8-bit resolution, 2 MSps). Two scenarios were evaluated:  
* (Scenario 1). An intruder in a coverage area where the legitimate aircraft is not present (intruder can be an aircraft with a fake or manipulated transponder or a nonauthorized ground transmitter);
* (Scenario 2). An intruder in a coverage area where the emulated aircraft is also present. The second type of attack is more difficult to contrast since the extracted signatures are derived from both the legitimate and the fake messages. The algorithm was evaluated, computing the Probability of False Alarm (the probability to have an alarm in case of legitimate signals) and the Probability of Detection (the probability to have an alarm in case of a real attack) and comparing the so-called Receiver Operating Characteristics (ROC) for different values of the threshold for the p-value of the KS-Test. Different sliding window sizes (\(M\)) and different values of \(K\) (K-NN parameter) were also tested. Finally, the computed results were compared with the ones obtained using the approach proposed in [31]. To compute the Probability of False Alarm, the recorded messages were assumed to be legitimate and the number of alarms over the number of tests was computed. To compute the Probability of Detection, the presence of an intruder was emulated by randomly selecting a signature coming from another aircraft; the number of alarms over the number of tests was then computed. In the first scenario, all the messages used to extract the SUT came from the randomly selected aircraft; in the second scenario, 50% of the messages came from the legitimated transmitter and the other 50% came from the randomly selected aircraft. Results for the first scenario, varying \(M\) and \(K\), are reported in Figure 6; all the signatures were computed using \(N=50\) consecutive ADS-B messages. Each point of each ROC curve represents the \(P_{fa}/P_{D}\) couple obtained for a given threshold on the p-value of the KS test. Each curve was obtained for a different configuration of parameters \(M\) and \(K\)  
Figure 6: Performance of the proposed method (Scenario 1) varying the alarm threshold. These curves show that it is possible to reach, at the same time, a very low probability of false alarm and a high probability of detecting the intruder. In general, the curves move to the left, increasing the size of the sliding windows (\(M\)) or the values of \(K\), with a small down-sliding effect. This means that by increasing the history, the performance in term of Probability of False Alarm increase (the region of main interest of the curve is near the elbow, where it is possible to obtain low \(P_{fa}\) and high \(P_{D}\) at the same time). In more detail, when increasing the value of \(K\), a more representative sample of the airplane signatures will be selected for the outlier test, thus improving the performance of the test--especially in terms of \(P_{fa}\), this can be seen in the figure, referring to the curve with the same history window size. On the other hand, it will also produce a small reduction in performance over the elbow of the curve, in term of \(P_{D}\). Concerning the history window size (\(M\)), as mentioned before, by increasing the size of this window, the performance of the algorithm improves but it also means an increase of the required computational capability.