# Metrics for Polyphonic Sound Event Detection
## 5 Choosing a Metric
### Measuring Performance of a System
#### 5.1.2 Event-Based Metrics  
Event-based F-score and error rate were calculated in two ways, first using the onset condition only, and second using both onset and offset conditions. The onset condition imposed the specific collar between the reference and system output events. The offset condition imposed either the same collar or a maximum difference between offsets of 50% of the reference event duration. The results are presented in Table 2.  
First of all, it can be noticed that the performance of the system measured using the event-based metrics is much lower than when using segment-based metrics. The event-based metrics measure the ability of the system of detecting the correct event in the correct temporal position, acting as a measure of onset/offset detection capability.  
Considering only the onset condition, the system is evaluated as having much better performance when the collar is larger. A larger collar behaves the same way as the longer segment in segment-based evaluation, allowing the system to produce a less exact output while still being considered correct. Figure 7 shows the class-wise F-score with 250 ms collar evaluated using onset only. It can be seen that for few event classes, the system is somewhat capable of detecting the onset within the allowed tolerance, while for other classes the system performs very poorly.  
\begin{table}
\begin{tabular}{c l c c c c c} \hline \hline
**Segment Length** & & **F** & **ER** & **Sensitivity** & **Specificity** & **ACC** & **BACC** \\ \hline
10 ms & class-based average & 34.5 & 0.87 & 27.3 & 99.5 & 96.7 & 63.4 \\  & instance-based average & **42.0** & **0.76** & **29.7** & 99.5 & 96.7 & **64.6** \\ \hline
1 s & class-based average & 44.9 & 0.89 & 43.2 & 97.1 & 92.4 & 70.1 \\  & instance-based average & **50.9** & **0.75** & **44.6** & 97.0 & 92.4 & **70.8** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Segment-based metrics calculated on the case study system.  
Figure 6: Segment-based F-score for the case study system, 10 ms evaluation segment.  
The offset condition allows a misalignment between the system output and ground truth event offsets. The permitted maximum misalignment is the collar length or 50% of the ground truth event length, whichever is higher. For very short events, this means that mostly the collar will be used, while long events benefit from the length condition. From Table 2 we can also notice that it is much easier to detect only onset of sound events compared to detecting both onset and offset. Similar behavior has been observed for algorithms used for note tracking in music transcription [30].  
The choice between a small and a large collar, and the use of onset only condition or onset/offset is dictated by the application--in applications that need a precise detection of the sound event boundaries, a short collar and both onset and offset conditions should be used. For applications where it is more important to detect the presence of certain sound events rather than their exact temporal position and duration, evaluation with a large collar is suitable, and the offset condition can be omitted. A larger collar is also more permissive on the subjectivity in onset/offset annotation.