# Interpretability of Methods for Switch Point Detection in Electronic Dance Music
## 4 Evaluation
### Evaluation Metrics  
Since SPD plays a central role in the creation of an automatic DJ, we put the emphasis on the precision of each method; that is, the ratio between the number of points estimated by the method that coincide with a human annotation (True Positive) over the total number of estimations returned by the method (i.e., True Positives and False Positives):  
\[Precision_{i}=\frac{TP_{i}}{TP_{i}+FP_{i}}.\]  
In this formula, an estimation is counted as a true positive if it lies within a 0.1 s window around an annotation (for more details, see mir_eval[32]). Precision is computed per track; a score of zero indicates that for the current track, the method produced no correct estimation, or no estimation at all. This effectively penalizes models that are not able to return at least one position per track since it makes those tracks unusable. The precision for a set of tracks is the average of the precision for each separate track; this metric is known as \"mean precision\":  
\[MeanPrecision=\frac{1}{n}\sum_{i=1}^{n}Precision_{i},\]  
The subscript \(i\) refers to the \(i\)-th track, while \(n\) is the total number of tracks (in the test set). \(TP_{i}\) and \(FP_{i}\) denote the number of True Positive and False Positive estimations in track \(i\), respectively. In addition to precision, it is common practice to evaluate methods according to their recall, which is the ratio between the number of estimations that coincide with an annotation (True Positive) and the total number of annotations (True Positive and False Negative). We argue that in the context of automatic mixing, recall is not as important as precision; that is, for an algorithm to be adequate, it is important to identify at least one viable switch point as opposed to all the possible ones. In general, the higher the recall (i.e., the more true positives for one given track), the more choices exist to create a transition. However, it is undesirable for a method to have a high recall if this comes at the cost of more false positive estimations. For these reasons, we consider mean precision as the mostinformative evaluation metric, and we provide the mean recall only as a complementary source of information. Our evaluation of the final decision (see Section 3 Methods) is conducted in isolation from the identification of the search space. This choice effectively eliminates the impact of errors that could occur in the identification of the offset and the core. In practice, the methods rely on an oracle that provides a search space which consists of manually annotated period boundaries from the start of the tracks until their core. The combined search space of the 150 tracks contains a total of 1948 candidate positions. Of these, 589 coincide with annotations. In the evaluation of the data-driven methods, we employed nested cross-validation with a 5-fold split of EDM-150 for the outer loop, and 4-fold split for the inner loop [33]. In the inner loop, the best hyper-parameters (HPs) are identified on a series of train and validation set splits. In the outer loop, the model performance (with the HPs identified) is evaluated on a series of (outer) train and test set splits. By doing the search of HPs in the inner loop, we ensure that the choice of the HPs does not overfit the test sets. By repeating the evaluation on five different test splits with the outer loop, we estimate the generalization capabilities of the methods and report the average score. For the evaluation of the experience-driven method, instead, we performed the evaluation only once on the whole dataset. Figure 3 presents the mean precision and the mean recall for all the methods; a commentary is provided in the following subsections, whereas a separate comparison that includes three baseline methods--\"All periods\", \"Mixed in Keys\", and \"DnB-autoDJ\"--appears in [2]. While nested cross-validation is used to evaluate the generalization capabilities of a method, it does not provide one final trained model.