# Deep Learning-Based End-to-End Carrier Signal Detection in Broadband Power Spectrum
## 4 Experiments
### Evaluation Results  
During the evaluation, we calculate the _F-score_ by the average precision rate (_AP_) and average recall rate (_AR_) to quantify and compare the performance of different trained models using the following formula:  
\[AP=\frac{\sum TP}{\sum TP+\sum TN} \tag{6}\]  
\[AR=\frac{\sum TP}{\sum TP+\sum FN} \tag{7}\]  
Figure 6: The training and validation loss of different scales of SCN, where \(L_{dd}\) is the total loss, \(L_{pd}\) is the PSD regression loss, and \(L_{bb}\) is the BW regression loss.  
Figure 7: The training and validation loss of the SCN-11x trained with different numbers of simulation samples, where \(L_{dd}\) is the total loss, \(L_{psd}\) is the PSD regression loss, and \(L_{bw}\) is the BW regression loss.  
\[F\text{-}Score=\frac{2*AP*AR}{AP+AR} \tag{8}\]  
where \(TP\) (true positive) denotes the number of subcarriers that are correctly detected, \(TN\) (true negative) denotes the number of subcarriers that are wrongly detected, and \(FN\) (false negative) denotes the number of subcarriers that are not detected, but are included in the test set. \(F\text{-}score\) is the harmonic means of \(AP\) and \(AR\) and is more reasonable than the arithmetic means [38].  
Figures A1-A8 show qualitative examples of the actual satellite broadband power spectrum set. All the subcarriers in these broadband power spectra have been correctly located and colored with a rectangle, which indicates that the proposed method is effective and achieves outstanding performance.  
* **SCN Model scale influence**  
With the application of CNN-based deep learning method in many fields, several studies [28; 39; 40; 41] show that the network scale (depth or layers) is significant for the performance of CNN. Moreover, the more layers used, the better the performance of the model. Furthermore, the different scales of SCN are validated on the test set with the number of residual layers ranging from 6 to 13.  
In Figure 8a, the \(F\text{-}score\) increases with the SCN model scale. Moreover, Figure 8b shows the precision-recall (PR) curves of these SCNs. Both \(AR\) and \(AP\) achieve high area under the curve (AUC) values when the network scale is beyond SCN-8x. The SCN-6x and SCN-7x do not have a good recall ratio, which leads to a deployment problem in real applications. The results indicate that the proposed method is effective and accurate.  
* **The effect comparison of the training set amounts**  
At present, deep learning is a data-driven science [10]. The more training samples used, the better the model would perform to a certain extent. Figure 9 shows the evaluation results of the SCN-11x trained with different numbers of training sets. We can see that the performance increases slightly as the training samples increase. On the basis of our analysis, all of the training samples are generated randomly through simulation, thus the diversity of the samples is insufficient. However, this also indicates that we do not have to use plenty of training samples to train the network for carrier signal detection in the broadband power spectrum. Furthermore, the fewer training samples used, the lower the cost of training time.  
Figure 8: The performance comparison of different scales of SCN. (**a**) \(F\text{-}scores\) change with training epochs. (**b**) PR curve.  
* **Complexity comparison** On the basis of the performance comparison of the SCN scale, we know that the model performs better with the SCN model scale. In comparison, the model complexity would also increase, as shown in Table 2. In practice, real-time is an essential factor in signal processing. A more complex model increases the processing time. Therefore, we need the most efficient model to balance the performance and processing time cost. Additionally, as can be seen in Table 2, the performance of SCN-\(11\times\) is as good as SCN-\(13\times\), but the inference time cost is 89.35% of SCN-\(13\times\), which is the most appropriate candidate.