# Metrics for Polyphonic Sound Event Detection
## 5 Choosing a Metric
### Comparing Performance of Different Systems  
For comparing performance of different systems for sound event detection, the metric should be chosen depending on the application and the cost associated to different types of errors. We give an example in Table 3. System A is the one presented in the previous subsection. System B is the baseline system provided in DCASE 2016 [22]--it uses mel-frequency cepstral coefficients and Gaussian mixture models in a binary classification setup, and is trained using the polyphonic audio. System C is the submission to DCASE 2103 [43]--it uses mel-frequency cepstral coefficients and hidden Markov models with multiple Viterbi passes for detection, and is trained using isolated examples of sounds for all the classes. For a complete comparison, performance of the systems is also compared to a zero-output system (frame-wise activity matrix containing only zeros), all-active-output system (frame-wise activity matrix containing only ones) and random-output system. The random-output system generates in each frame for each event an activity level (0 or 1) based on the average sparseness of the dataset (priors 0.95 and 0.05 for 0 and 1, respectively). The random-output system was simulated 1000 times and the average calculated values over all runs are presented. The results in Table 3 are obtained using the same data as in Section 5, with one second evaluation segment and 250 ms collar.  
Faced with a choice between these systems, if we want a system that finds best the regions where certain event classes are active, we would choose system B for the 68.5% segment-based F-score. The system also makes less errors in the evaluated segments than both systems A and C, and is therefore a good choice for detecting most regions with active events. However, if we want a system that detects best the position in time of event instances, we would choose system A based on its 30.8% event-based F-score. This system makes approximately 1.5 times more mistakes than the number of events to be detected, while correctly detecting approximately one third of them. The event-based performance of system C is comparable with the performance of system A when using only the onset condition, but system C is much better at detecting the sound event offsets, with a 20.8% F-score. A  
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Compared Systems** & \multicolumn{2}{c}{**Segment-Based**} & \multicolumn{2}{c}{**Event-Based**} & \multicolumn{2}{c}{**Event-Based**} \\  & & \multicolumn{2}{c}{**Onset Only**} & \multicolumn{2}{c}{**Onset + Offset**} \\  & **F** & **ER** & **F** & **ER** & **F** & **ER** \\ \hline system A (NMF [28]) & 42.0 & 0.76 & **30.8** & 1.49 & 10.1 & 2.06 \\ system B (GMM [22]) & **68.5** & **0.49** & 10.7 & **1.42** & 4.4 & **1.62** \\ system C (HMM [43]) & 46.9 & 0.87 & 30.0 & 1.47 & **20.8** & 1.76 \\ \hline zero-output system & 0.0 & 1.00 & 0.0 & **1.00** & 0.0 & **1.00** \\ all-active-output system & 15.4 & 10.93 & 0.0 & 1.44 & 0.0 & 1.44 \\ random-output system & 9.0 & 1.53 & 1.6 & 2.59 & 1.1 & 2.67 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparing systems using different metrics calculated using instance-based averaging, 1 s segment, 250 ms collar.  
choice of system based on ER would indicate system B as the best, making the smallest amount of errors in both segment-based and event-based calculation.  
The zero output system has an F-score of 0 and ER of 1, as expected. The all-active output system achieves a nonzero segment-based F-score by finding all the true positives. It is however penalized by introducing false positives on all other instances in the event activity matrix. In event-based evaluation, this system outputs one long event active for the whole duration of the test audio, which counts as an insertion, with no correct output. The random-output system obtains a performance between the two extremes--but still very low in comparison with all the systems based on learning.