# V2X Wireless Technology Identification Using Time-Frequency Analysis and Random Forest Classifier
## 5 Performance Evaluation
### Simulation Results
#### 5.2.2 Performance Analysis  
The proposed approach, consisting of IF features combined with the random forest classifier, is evaluated in tandem with that based on SCF features and the SVM classifier. Each of the datasets is shuffled and split into training and test sets containing 3000 and 1500 samples, respectively. Then, the input data is normalized and scaled, which is a crucial step to alleviate the effect of SNR variations, especially for distance-based classifiers like SVM. Implementation and evaluation are conducted in the open source Scikit-learn software library [39]. The confusion matrices of the two considered techniques are depicted in Figure 4. They show that the SCF features provide a slightly worse performance compared to the IF features when dealing with LTE-V2X and NR-V2X signals. Indeed, the two technologies share many PHY layer parameters, as previously seen in Section 3. Since SCF reveals the hidden periodicities within the signals, which are caused by the symbol period and cyclic prefix duration among others, their similarity decreases the distance between samples and prevents the SVM algorithm from correctly identifying those signals. Figure 3: t-sne representation of (**a**) SCF and (**b**) IF features. Their identification rates are 96% and 91%, respectively. The difference between these two rates flows from the fact that the NR-V2X standard has more configurations, and thus more dispersed SCF values, than the LTE-V2X standard. Therefore, the boundary placed by the SVM classifier to identify the NR-V2X signals is less accurate than that of the LTE-V2X signals. On the other hand, the random forest classifier increases the identification rate to 99% for both signals because the confusion between them has been decreased by using IF features. Their relevance comes from the power of intrinsic mode functions and instantaneous frequency to bring out the local time-frequency characteristics of the signals. When it comes to the 802.11p technology, its unique characteristics make it more distinguishable, and both cyclostationarity and time-frequency based features can be used to identify ITS-G5 signals with an accuracy of 100%. The precision, recall and F1-score of the three signal types and both techniques are summarized in Table 2. A simple comparison shows that the classification results are in line with the previous t-sne analysis. However, they only represent the global performance of the algorithms within a wide range of SNRs and do not really reflect the impact of this parameter on the identification accuracy. In order to investigate this relationship, the two classifiers are trained and tested on signals of each SNR level separately. Figure 5 depicts the accuracy variation of both techniques, along with that obtained by implementing the classification algorithm in [12], taken as an example of a comparison with the statistical approach. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **Signal** & **Precision** & **Recall** & **F1–Score** \\ \hline \multirow{4}{*}{Proposed IF with RF} & ITS-G5 & 1 & 1 & 1 \\  & LTE-V2X & 0.99 & 0.99 & 0.99 \\  & NR-V2X & 0.99 & 0.99 & 0.99 \\  & Average & 0.99 & 0.99 & 0.99 \\ \hline \multirow{4}{*}{SCF with SVM [20]} & ITS-G5 & 1 & 1 & 1 \\  & LTE-V2X & 0.91 & 0.96 & 0.94 \\ \cline{1-1}  & NR-V2X & 0.96 & 0.91 & 0.93 \\ \cline{1-1}  & Average & 0.96 & 0.96 & 0.96 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Performance metrics of random forest and support vector machine classifiers. Figure 4: Confusion matrix of (**a**) SCF with SVM and (**b**) IF with random forest techniques. The results show that the cyclostationarity-based features are more sensitive to the noise level than the proposed IF features. For instance, the SVM classifier exceeds 90% accuracy at \(-4\) dB, then gives the best performance, 100%, at \(-2\) dB and remains constant until 18 dB. This behavior can be explained by the decrease of SCF amplitudes at low SNRs.