# A Deep Learning Framework for Signal Detection and Modulation Classification
## 4 Results
### Performance on Modulation Recognition  
For signal modulation recognition, we set a series of experiments to test the network performances. Figure 13a shows the recognition performances of each modulated signal under the different SNR. It can be seen that the algorithm can still achieve better performance when the SNR is very low. Because its modulation complexity, the performance of 64 QAM signal is worse than other signals, but it still can achieve 94% accuracy at 7 dB. For BPSK and OQPSK signals, they have distinct visual characteristics from other modulated signals in the eye diagram and the vector diagram, which recognition accuracy can reach 100% even in 0 dB. And it is also obvious that the recognition performance of circular modulation signals [8 PSK, 16 APSK, 32 APSK] is better than QAM modulation signal. To understand the results better, the confusion matrices in different SNR levels are presented in Figure 13b-d. It can be seen that the network shows excellent performance in discriminating BPSK, QPSK, OQPSK, 8 PSK, and 16 APSK. Moreover, in our experiments, it can be seen that 16 QAM is more likely confused with 64 QAM, while 16 APSK is more likely confused with 64 QAM. Figure 12: Different networks performances. (**a**) performances for mAP; (**b**) performances for time. Figure 13: _Cont._For accuracy comparison, we consider four different modulation classification algorithms. 1. [leftmargin=*]
2. Cumulant: A traditional signal processing algorithm using the fourth-order cumulant C40 as the classification statistics [19]. 3. SVM-7: An ML-based algorithm using the SVM with seven features, including three fourth-order cumulants C40, C41, and C42 and four sixth-order cumulants C60, C61, C62, and C63 [20]. 4. CNNs for IQ waveform: A DL-based algorithm using the CNNs with the signal IQ waveform [4]. 5. CNNs for constellation: An DL-based algorithm using the CNNs with the signal constellation [35]. Figure 14 presents the average classification accuracy of five algorithms versus SNR. The average accuracy is obtained by averaging the classification performance of eight modulation categories. The performance results of our algorithm outperform all other algorithms. Figure 14: Different methods performance versus SNR. Figure 13: The performance of each modulation (**a**) classification accuracy for each modulation versus SNR; (**b**) normalized confusion matrix in 0 dB; (**c**) normalized confusion matrix in 3 dB; (**d**) normalized confusion matrix in 10 dB  
Considering the error of the carrier frequency estimation by SSD networks and FFT in practice, we research the network recognition performance in different frequency offsets. We set a series of frequency offset for signals, and the result is shown in Figure 15. It can be seen that the recognition accuracy of the signals with a frequency offset is lower than those without frequency offset. When the signals have a large frequency offset, the network is no longer suitable. We also collect some signals from a real satellite communication system, and the real-time wireless channel is performed in the received signals. And then, we use a signal playback device, a DSP card, and PCs to simulate signal reception process. From Figure 15a, we can obtain that the recognition accuracy on real data is lower on simulated data at same SNR level. It may be due to the training data, which not consider the actual channel environment clearly. But the recognition accuracy can still reach 90% when the SNR is 4 dB. And for further research, we will make full use of the real signal to make our model more robust. We also consider the influence of the symbol numbers and the eye number in eye diagram on the network performance. We obtain the best parameter settings of samples by grid search. The symbol number is set as 200, 400, 800, and 1000, respectively, while the eye number is set as 2, 3, 4, and 5. The results are shown in Figure 16. It can be seen that theses parameters do affect network performance.