# Metrics for Polyphonic Sound Event Detection
## 5 Choosing a Metric
### Measuring Performance of a System
#### 5.1.3 Discussion  
The results from Tables 1 and 2 show the difficulty of characterizing the system performance through a single metric. It is important to know that there is always a trade off between measured aspects, and make a choice of metric based on what the requirements for the system are.  
F-score is the first choice metric in many cases due to its widespread use in other research areas. From Table 1 we can conclude that when measured in one second segments, half of the activity of sound events has been correctly detected by the presented system. However, the system still makes many errors, indicated by ER. Error rate gives the measure of how much the system is wrong. In the provided example, the system makes a number of errors that is 75% of the number of events to be detected, which is relatively high.  
Accuracy gives a measure of how well the system output matches the reference in both true positives and true negatives. The example system outputs a very small amount of false positives, which results in a very good rate of true negatives. Combining this with the fact that the used data has relatively sparse events' activity and in fact there are not too many positives to be detected, the system has an accuracy close to 100%. The ability to correctly output true negatives is important in applications where false positives are highly undesired. For example in an automatic alert system  
\begin{table}
\begin{tabular}{l l c c c c} \hline \hline  & & \multicolumn{2}{c}{**Onset Only**} & \multicolumn{2}{c}{**Onset + Offset**} \\
**Collar** & & **F** & **ER** & **F** & **ER** \\ \hline
100 ms & class-based average & 7.5 & 2.10 & 4.7 & 2.10 \\  & instance-based average & **20.7** & **1.74** & **8.2** & **2.11** \\ \hline
250 ms & class-based average & 10.5 & 2.25 & 4.1 & 2.10 \\  & instance-based average & **30.8** & **1.49** & **10.1** & **2.06** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Event-based F-score and ER calculated on the case study system.  
Figure 7: Event-based F-score for the case study system, 250 ms collar, onset only.  
in an environment with sparse events, false positives will have immediate negative effect on user satisfaction. For such applications, specificity is a more appropriate measure, or true negative rate. On the other hand, there are monitoring applications in which it is more important to have very low rate of false negatives, such as gun shot detection or car crash detection. In this case the best choice of measure would be recall (sensitivity). However, if general accuracy is of interest for the application, balanced accuracy is a better choice for metric, because the true negatives have less weight in it.  
Event-based metrics illustrate better the ability to correctly locate and label longer blocks of audio. For application where detection of correct onsets (and offsets) of sounds is important, the metric of choice should be event-based. The event-based error rate of the example system as seen in Table 2 is over 1, which means that the system makes much more errors than correct outputs. According to Equation (5), it is always possible to obtain an error rate of 1 by using a system that does not output anything. However, even with an error rate over 1, there are obviously many events correctly detected, according to F-score being 30.8%.