Package: sed-scores-eval
sed_scores_evalsed_scores_eval is a package for the efficient (threshold-independent)
evaluation of Sound Event Detection (SED) systems based on the SED system's
soft classification scores as described inThreshold-Independent Evaluation of Sound Event Detection ScoresJ. Ebbers, R. Serizel and R. Haeb-Umbachin Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) 2022https://arxiv.org/abs/2201.13148With SED systems providing soft classification scores (usually frame-wise),
performance can be evaluated at different operating points (OPs) by varying the
decision/discrimination threshold used for binarization of the soft scores.
Other evaluation frameworks evaluate a list of detected sounds
(list of event labels with corresponding event onset and offset times) for each
decision threshold separately.
Therefore, they can not be used to accurately evaluate performance curves over
all thresholds (such as Precision-Recall curves and ROC curves) given that
there are many thousands (or even millions) of thresholds (as many as there are
frames in the dataset) that result in a different list of detections.
Performance curves can at most be approximated using a limited subset of
thresholds which, however, may result in inaccurate curves (see Figure below).
sed_scores_eval, in contrast, efficiently evaluates performance for all
decision thresholds jointly (also for sophisticated collar-based and
intersection-based evaluation criteria, see paper for details). It therefore
enables the efficient and accurate computation of performance curves such as
Precision-Recall Curves and ROC Curves.If you use this package please cite our paper.Supported Evaluation CriteriaIntermediate Statistics:Segment-based[1]: Classifications and targets are defined and
evaluated in fixed length segments.Collar-based (a.k.a. event-based)[1]: Compares if detected event
(onset, offset, event_label) matches a ground truth event up to a certain
collar on onset and offset.Intersection-based[2]: Evaluates the intersections of detected and
ground truth events (Please also cite[2]if you use intersection-based
evaluation)Evaluation Metrics / Curves:Precision-Recall (PR) Curve: Precisions for arbitrary decision thresholds
plotted over RecallsF-Score Curve: F-Scores plotted over decision thresholdsF-Score @ OP: F-Score for a specified decision thresholdF-Score @ Best: F-Score for the optimal decision threshold (w.r.t. to the
considered dataset)PSD-ROC Curve: effective True Positive Rates (eTPRs) plotted over effective
False Positive Rates (eFPRs) as described in[2].PSD Score (PSDS): normalized Area under PSD-ROC Curve (until a certain
maximum eFPR).(Please also cite[2]if you use PSD-ROC and/or PSDS)[1]A. Mesaros, T. Heittola, and T. Virtanen,
"Metrics for polyphonic sound event detection", Applied Sciences,
2016,[2]C. Bilen, G. Ferroni, F. Tuveri, J. Azcarreta and S. Krstulovic,
"A Framework for the Robust Evaluation of Sound Event Detection",
in Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP),
2020,
arXiv:https://arxiv.org/abs/1910.08440IPython NotebooksHave a look at the provided examplenotebooksfor usage example
and for some comparisons/validations w.r.t. reference packages.Input FormatSystem's Classification ScoresThe system's classification scores need to be saved in a dedicated folder with
a tsv score file for each audio file from the evaluation set.
The score files have to be named according to the audio file names.
If, e.g., the audio file is "test1.wav" the score file's name needs to be
"test1.tsv".
For each score window (arbitrary and also varying window lengths are allowed
but windows need to be non-overlapping and gapless, i.e., the onset time of the
next window must be the offset time of the current window) the onset and offset
times of the window (in seconds) must be stated in the first and second column,
respectively, followed by classification scores for each event class in a
separate column as illustrated in the following example:onsetoffsetclass1class2class3...0.00.020.0105350.0575490.063102...0.020.040.0011960.1677300.098838.....................4.764.780.0151280.7696870.087403...4.784.80.0020320.5875780.120165.....................9.9810.00.0314210.0897160.929873...At inference time, when your system outputs a classification score arrayscores_arrof shape TxK with T and K being the number of windows and event
classes, respectively, you can conveniently write the score file of above
format as follows:sed_scores_eval.io.write_sed_scores(scores_arr,'/path/to/score_dir/test1.tsv',timestamps=timestamps,event_classes=event_classes)wheretimestampsmust be a 1d list or array of length T+1 providing the
window boundary times andevent_classesmust be a list of length K providing
the event class names corresponding to the columns inscores_arr.In case the output scores of the whole dataset fit into memory, you can also
provide a dict of pandas.DataFrames of above format, where dict keys must be
the file ids (e.g. "test1").
Score dataframes can be obtained from score arrays analogously to above byscores["test1"]=sed_scores_eval.utils.create_score_dataframe(scores_arr,timestamps=timestamps,event_classes=event_classes)Ground TruthThe ground truth events for the whole dataset must be provided either as a
file of the following formatfilenameonsetoffsetevent_labeltest1.wav3.984.86class2test1.wav9.0510.0class3test2.wav0.04.07class1test2.wav0.08.54class2test2.wav5.437.21class1............or as a dict{"test1":[(3.98,4.86,"class2"),(9.05,10.0,"class3")],"test2":[(0.0,4.07,"class1"),(0.0,8.54,"class2"),(5.43,7.21,"class1")],...}which can be obtained from the file byground_truth_dict=sed_scores_eval.io.read_ground_truth_events(ground_truth_file)Audio durationsIf required, you either have to provide the audios' durations (in seconds) as a
file of the following format:filenamedurationtest1.wav10.0test2.wav9.7......or as a dict{"test1":10.0,"test2":9.7,...}which can be obtained from the file bydurations_dict=sed_scores_eval.io.read_audio_durations(durations_file)InstallationInstall package directly$pipinstallgit+https://github.com/fgnt/sed_scores_eval.gitor clone and install (editable)$gitclonehttps://github.com/fgnt/sed_scores_eval.git
$cdsed_scores_eval
$pipinstall--editable.