# Deep-Learning-Based Stream-Sensing Method for Detecting Asynchronous Multiple Signals
## 6 Evaluation
### Performance Metrics  
For performance evaluation in the rescue scenario, the following relaxed performance metrics are introduced:  
* Precision with \(d\) for \(k\) transmitted signals (\(P_{k\pm d}\)): the relaxed precision is defined as follows. \[P_{k,d}=\frac{TP_{k\pm d}}{TP_{k\pm d}+FP_{k\pm d}},\] (11) where \(TP_{k\pm d}\) and \(FP_{k\pm d}\) are true positive and false positive, respectively. The parameter \(TP_{k\pm d}\) denotes the case when a scheme predicts \(k\) transmitted signals; the actual number of transmitted signals is within \([k-d,k+d]\). The parameters \(FP_{k\pm d}\) represent the case that when the detected number of transmitted signals is \(k\), the actual number of transmitted signals is not within \([k-d,k+d]\).
* Recall with \(d\) for \(k\) transmitted signals (\(R_{k\pm d}\)): the relaxed precision is defined as follows. \[R_{k,d}=\frac{TP_{k\pm d}}{TP_{k\pm d}+FN_{k\pm d}},\] (12) where \(FN_{k\pm d}\) is false negative and includes the case that when the actual number of transmitted signals is \(k\), the detected number of transmitted signals is not within \([k-d,k+d]\).  
Table 2 shows how to set \(TP_{k\pm d}\), \(FP_{k\pm d}\), and \(FN_{k\pm d}\) for \(k=2\), \(d=1\).  
The recall \(R_{k,d}\) is within \([0,1]\). As \(R_{k,d}\) becomes closer to 1, the detection performance of a scheme becomes better. In general, \(R_{k,d}\) is an index indicating how well the detected value is within \([k-d,k+d]\) when the actual number of transmitted signals is \(k\). Hence, the higher the recall value, the better the detection performance of a detection scheme.  
The precision \(P_{k,d}\) is also within \([0,1]\). As \(P_{k,d}\) becomes closer to 1, the reliability of a scheme becomes better. For example, if \(P_{2,1}=1\), it means that for the predicted \(k=2\) transmitted signals, and then the actual number of transmitted signals is within \([1,3]\) with probability of 1. On the other hand, for \(P_{2,1}=0\), although the detected number is \(k=2\), the actual number of transmitted signals is not within \([1,3]\) with a probability of 0. As a result, the precision is an indicator of how reliable the predicted results are.  
For this work, the reason why we have introduced relaxed performance metrics for the performance evaluation can be explained as follows. In a rescue situation, it might be more important to provide a rough overview of the situation without critical errors. In other words, even if the predicted value is slightly inaccurate, it can still be considered meaningful enough as the predicted value is reasonably close to the actual value from a contextual point of view. Note that for \(k\in[0,2]\), we show the precision and recall with only \(d=0\) since the error of 1 is relatively large compared to the actual values 0, 1, and 2. Furthermore, in computer vision, relaxed performance metrics are also used as the standard recall-at-k metric to evaluate how well a proposed system predicts values as close as possible to the true values [52; 53; 54].  
In this paper, by comparing those results with \(d\in\{0,1\}\), we verify that our scheme with only the minimal relaxation value (\(d=1\)) provides a reliable performance in various SNR scenarios while outperforming the existing solutions, AIC and MDL. Based on the results, our ML-based stream-sensing scheme shows high precision and recall performance with \(d=1\). Hence, we confirm that our proposed scheme can detect the number of transmitted signals with very little error.