# Metrics for Polyphonic Sound Event Detection
## 4 Metrics for Polyphonic Evaluation
### Other Metrics  
_Sensitivity and specificity_ represent the true positive rate and true negative rate, respectively. They are used as a pair, to illustrate the trade-off between the two measured components.  
\[Sensitivity=\frac{TP}{TP+FN}\]  
\[Specificity=\frac{TN}{TN+FP} \tag{6}\]  
A related measure is given by the Receiver Operating Characteristic (ROC) curve and Area Under the Curve (AUC). A ROC curve plots true positive rate _vs._ false positive rate, _i.e._, sensitivity vs (1-specificity) at various thresholds. AUC allows summarizing the ROC curve into a single number,and is often used for comparing performance of binary classifiers. AUC was used to evaluate sound classification performance in [25] for a multi-label task by using binary classifiers for each class and calculating the average of the class-wise AUC values.  
_Accuracy_ measures how often the classifier takes the correct decision, as the ratio between the number of correct system outputs and the total number of outputs.  
\[ACC=\frac{TP+TN}{TP+TN+FP+FN} \tag{7}\]  
Accuracy does not characterize well the ability of the system to detect the true positives: when the events' activity is sparse, the count of true negatives will dominate the accuracy value. It is possible to assign weights to \(TP\) and \(TN\), obtaining the _balanced accuracy_:  
\[BACC=w\cdot\frac{TP}{TP+FN}+(1-w)\cdot\frac{TN}{TN+FP} \tag{8}\]  
With equal weights (\(w=0.5\)), the balanced accuracy is the arithmetic mean of sensitivity and specificity. Specificity, accuracy and balanced accuracy are not defined in the event-based calculation, as there is no available count of true negatives.  
Another definition for accuracy has been proposed in [37] for music transcription, and can be easily generalized for both segment-based and event-based sound event detection cases. It was defined as:  
\[ACC_{MIR}=\frac{TP}{TP+FP+FN} \tag{9}\]  
by discarding \(TN\) from Equation (7). A related metric is _Mean Overlap Ratio_[38] that was used to measure the average temporal overlap between correctly transcribed and reference notes. This is a temporal measure of accuracy as defined by Equation (9), as it measures the amount of overlap time--true positive duration--w.r.t. combined duration of the reference and output event--which for misaligned onsets and offsets consists of false negative, true positive and false positive durations. Mean overlap ratio for missed or inserted events is 0, leading to very low overall average for systems that output many errors, as is the case with polyphonic sound event detection systems at the moment.  
Another accuracy measure is the _Relative Correct Overlap_ used in chord transcription, measuring the proportion of the system output that matches the reference as relative frame count [39] or in seconds [40]. Because chords are combinations of individual notes, this metric measures frame-based accuracy of certain combinations of classes. In polyphonic sound event detection, this measure would represent the proportion of frames or time when the system detects correctly all events active in the reference. However, combinations of sound events have no specific meaning for the detection task, therefore such a measure is not conceptually meaningful for it.  
Accuracy variants defined by Equations (7)-(9) have the advantage of being bounded between 0 and 1, with 1 in the case when the system makes no errors. The expressions in Equations (7) and (8) reach 0 when the system output contains only errors, while Equation (9) is 0 also when the system outputs nothing, as its value is not linked to the true negatives. Accuracy does not provide any information on the trade-off between the error types, which may be considered a disadvantage in certain situations. At the same time, it is capable of counting only insertions and deletions, while in some applications it is preferable to count substitutions.