# Music Similarity Detection Through Comparative Imagery Data
## 4 Experimental Results: Model Training, Validation, and Testing
### Machine Learning Experiments  
Like many ML workflows, we conducted numerous experiments to optimize various hyperparameters. For the four sub-models corresponding to the four features identified (Section 3), Table 7 shows the best combination obtained for training each of the four sub-models.  
The results of the experiments showed that Earth mover's distance calculated with respect to the L1 norm performed the best with the three histogram features, and L1 norm worked the best for the feature of distance between the two most common vertical intervals. Moreover, since the distance between the two most common vertical intervals is a scalar feature, Earth mover's distance is not meaningfully applicable and was not considered as an option for this feature.  
The individual validation performance of each feature-based sub-model is also summarized in Table 7, along with chosen training hyperparameters. Figure 10 shows that all models clearly train properly and converge to optimal parameters within the number of epochs allocated to training in each case. Overfitting is also demonstrably avoided, which is expected due to the use of an L2 regularizer and dropout.  
Lastly, the confusion matrix of each CNN is given in Figure 11, where the accuracy measures of the four sub-models in the validation stage are displayed. Each sub-model can separate between classes almost perfectly, which can be inferred from the area under the receiver operator characteristic curve (AUC). All features have an almost perfect AUC score and a negligible classification error. However, this almost perfect performance does not mean that (i) all four sub-models are in agreement for every pixel location in a set of  
\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline
**Feature** & **Metric** & **Loss** & **F1 Score** & **Accuracy** & **AUC** & **LR** & **Epochs** \\ \hline _MI Histogram_ & Earth Mover’s Distance & 0.08 & 0.90 & 0.97 & 0.99 & 0.005 & 300 \\ \hline _FFPC Histogram_ & Earth Mover’s Distance & 0.10 & 0.94 & 0.98 & 0.99 & 0.005 & 100 \\ \hline _WVI Histogram_ & Earth Mover’s Distance & 0.08 & 0.99 & 1.00 & 1.00 & 0.01 & 100 \\ \hline _DBTMCVI Heatmap_ & L1 Norm & 0.12 & 0.94 & 0.98 & 0.99 & 0.01 & 200 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Summary of model performance with respect to different combinations of the specified hyperparameters using validation metrics. The abbreviations of the feature names are MI (melodic interval), FFPC (folded fifths pitch class), WVI (vertical interval), and DBTMCVI (distance between two most common vertical intervals).  
feature-based input heatmaps, and (ii) their performance in independent testing is not assured since the sub-models are not trained with any legally verified data.  
The performance of the ensemble model on the same validation set used in validation-testing the individual sub-models is better than the individual performance of each sub-model in terms of all validation metrics as shown in Table 8. The confusion matrix in in Figure 12 also shows that the ensemble model has achieved a faultless classification of the heatmaps in the validation set. Therefore, enabling different feature-specific sub-models to make collective decisions is clearly beneficial to similarity detection.  
Figure 11: Confusion matrices of the validation set for all of the features with the final trained models: (**a**) melodic interval histogram; (**b**) folded fifths pitch class histogram; (**c**) wrapped vertical interval histogram; (**d**) distance between two most common vertical intervals.  
Figure 10: Plots of training and validation loss against the number of iterations for all of the features.