# Metrics for Polyphonic Sound Event Detection
## 4 Metrics for Polyphonic Evaluation
### Error Rate  
Error rate measures the amount of errors in terms of _insertions_ (I), _deletions_ (D) and _substitutions_ (S). To calculate segment-based error rate, errors are counted segment by segment. In a segment \(k\), the number of substitution errors \(S(k)\) is the number of reference events for which a correct event was not output, yet something else was. This is obtained by pairing false positives and false negatives, without designating which erroneous event substitutes which. The remaining events are insertions and deletions: \(D(k)\)--the number of reference events that were not correctly identified (false negatives after substitutions are accounted for) and \(I(k)\)--the number of events in system output that are not correct (false positives after substitutions are accounted for). This leads to the following formula:  
\[ \begin{split}& S(k)=min(FN(k),FP(k))\\ & D(k)=max(0,FN(k)-FP(k))\\ & I(k)=max(0,FP(k)-FN(k))\end{split}  \tag{4}\]  
Figure 4: Calculation of segment-based metrics. Figure 5: Calculation of event-based metrics. Total error rate is calculated by integrating segment-wise counts over the total number of segments \(K\), with \(N(k)\) being the number of sound events marked as active in the reference in segment \(k\):  
\[ER=\frac{\sum_{k=1}^{K}S(k)+\sum_{k=1}^{K}D(k)+\sum_{k=1}^{K}I(k)}{\sum_{k=1}^{ K}N(k)} \tag{5}\]  
The calculation is illustrated in Figure 4, panel B. A similar calculation of error rate for polyphonic detection was used in evaluating multi-pitch transcription in MIREX [30] and sound event detection in DCASE 2013 [20]. The event-based error rate is defined with respect to the sound events, by taking into account temporal position and label of each sound event in system output compared to the ones in the reference. Sound events with correct temporal position but incorrect class label are counted as substitutions, while insertions and deletions are the sound events unaccounted for as correct or substituted in system output or reference, respectively. Overall error rate is calculated according to Equation (5) based on the error counts. This calculation is illustrated in Figure 5, panel B. Class-wise measures of error rate (segment or event based) aggregate error counts individually for each sound event class, therefore they cannot account for substitutions, but can count only insertions and deletions. In other specific sound recognition areas, error rate is defined differently to reflect the target of the evaluation. In speech recognition, _Word Error Rate_ is defined with respect to word sequences, defining an insertion as a word present in the output but not in the reference, a deletion as a word present in the reference but not in the output, and a substitution as a word in the reference erroneously recognized (different word in the output). In practice, one substitution represents one insertion and one deletion (one word is missed, another inserted). Because \(N\) is the number of words in the reference, the word error rate can be larger than 1.0. In speaker diarization, the error rate measures the fraction of time that is not attributed correctly to a speaker or to non-speech. Temporal errors are: \(S\)--the percentage of scored time that a speaker ID is assigned to the wrong speaker, \(I\)--the percentage of scored time that a non-speech segment from the reference is assigned to a hypothesized speaker, and \(D\)--the percentage of scored time that a speaker segment from the reference is assigned to non-speech. The total error is measured as the fraction of temporal error with respect to the total time of reference speech. This error rate was used for evaluating sound event detection in [29], after the multi-label reference was projected to single label for comparison with single label system output. The temporal error was calculated as in speaker diarization, with event classes instead of speaker ID.