# Metrics for Polyphonic Sound Event Detection
###### Abstract  
This paper presents and discusses various metrics proposed for evaluation of polyphonic sound event detection systems used in realistic situations where there are typically multiple sound sources active simultaneously. The system output in this case contains overlapping events, marked as multiple sounds detected as being active at the same time. The polyphonic system output requires a suitable procedure for evaluation against a reference. Metrics from neighboring fields such as speech recognition and speaker diarization can be used, but they need to be partially redefined to deal with the overlapping events. We present a review of the most common metrics in the field and the way they are adapted and interpreted in the polyphonic case. We discuss segment-based and event-based definitions of each metric and explain the consequences of instance-based and class-based averaging using a case study. In parallel, we provide a toolbox containing implementations of presented metrics.  
pattern recognition; audio signal processing; audio content analysis; computational auditory scene analysis; sound events; everyday sounds; polyphonic sound event detection; evaluation of sound event detection +
Footnote †: journal:  
both class name and temporal information. We use the term _polyphonic annotation_ to mark annotations containing sounds that overlap in time.  
There is no universally accepted metric for evaluating polyphonic sound event detection performance. For a monophonic annotation and monophonic output of sound event detection system, the system output at a given time is either correct or incorrect if the predicted event class coincides or not with the reference class. In polyphonic sound event detection, the reference at a given time is not a single class, and there can be multiple correctly detected and multiple erroneously detected events at the same time, which must be individually counted. A similar situation is encountered in polyphonic music transcription [21], where at a given time there can be overlapping notes played by different instruments.  
In this paper, we review the use of metrics for measuring performance of polyphonic sound event detection, based on adapting metrics from neighboring fields to cope with presence of multiple classes at the same time. Special sessions at recent conferences and the AASP Challenge on Detection and Classification of Acoustic Scenes and Events 2013 [20] and 2016 [22] underline the acute need for defining suitable evaluation metrics. We introduce and analyze the most commonly used metrics and discuss the way they are adapted and interpreted in the case of polyphonic sound event detection. We also discuss the different possibilities for calculating each metric, and their advantages and disadvantages.  
This paper is organized as follows: Section 2 presents a background for sound event detection and classification, and the components of a sound event detection system. Section 3 presents definitions of intermediate statistics that are used for computing all metrics, and Section 4 reviews the metrics. Section 5 discusses the choice of metrics when evaluating a sound event detection system and comparing performance of different systems, using a case study example. Finally, Section 6 presents conclusions and future work.