# Interpretability of Methods for Switch Point Detection in Electronic Dance Music
## 4 Evaluation
### Nlc-Gbdt  
This method achieves the highest precision (93%) among all methods considered in this study (by contrast, its recall is rather low, 29%). We attribute these results to the fact that this method uses a threshold as a decision rule to select what features are relevant in SPD. The threshold is learned to maximize the mean precision on the validation data, and a very high value is selected so that only one position (the one with the highest probability) is returned for each track. The interpretation of the model is in itself a challenge, as too many trees are generated. Instead, we employ treeExplainer [35], a Python 3 library to estimate SHAP (SHapley Additive exPlanations) values for tree models and ensembles of trees. These values measure the contribution of each feature to the prediction of the model for each sample of the training data. Then, by combining all the local SHAP values, the global explanation of the model is computed and represented with a summary plot, as shown in Figure 6. This global explanation represents the magnitude, prevalence, and direction of the effect for each feature. This plot is comparable to the features' coefficients learned by the linear classifier, but it singles out the contribution of each feature between the prevalence (number of samples) and magnitude (SHAP value) of its effect. In this plot, each feature is represented  
Figure 5: Decision tree trained on the whole EDM-150. The root node represents the initial distribution of the ground truth labels. Traversing down the tree, the candidates are divided according to the learned decision rules toward the child nodes until reaching the leaves of the tree. on a different row, ordered from top to bottom according to their importance (average influence over every sample). Each dot represents a sample, its color corresponds to the value of the feature for this observation, and its horizontal position represents the SHAP value--the effect that the feature has on the prediction for that sample (i.e., how much the feature changes the output probability of the model). Dots with identical SHAP values are stacked to display the distribution of the effects of the features. On the one hand, for the most part, the summary plot confirms our intuition about the relevance of some features for SPD described in Section 2.2. Specifically, novelty in energy (energy CK), in harmony (CQT CK), and in timbre (MFCC CK) have the strongest effects in this model, whereas novelty in PCP (PCP CK), and repetition of MFCC (MFCC SF) are the least influential features--confirming that pitch distribution is not homogeneous in a segment and that timbre does not display a highly repetitive behavior. On the other hand, the bass drum novelty (Bass drum CK) has a moderate effect, lower than our expectation. While this is not a conclusive explanation, we identify two possible reasons for the limited importance of this feature. First, bass drum novelty is heavily correlated to energy novelty. Since the model will use both features to capture the same effect, the influence of each one on the final decision is smaller than if only one feature were provided. Second, we note that the bass drum onsets are not accurately estimated, see score for \"BD\" [28] (p. 6), thus adding noise to the feature and reducing its influence. In the summary plot, one can notice samples with the same feature value (color) but different SHAP values (X-position), an effect due to the interactions between features. Such a variation can be shown in a so-called dependence plot, as represented in Figure 7a for the energy novelty. This plot illustrates that when the value of the feature increases, the probability that the position is a switch point increases logarithmically with a wide variance (vertical dispersion) due to the interaction with other features. This variance is explained, at least partially, by the interaction with timbre novelty, whose value is represented by the color of the dots (Figure 7a).