# Music Similarity Detection Through Comparative Imagery Data
## 2 Methods
### Method 3: Image-Based Machine Learning  
Following our experimentation on the 33 features (see Section 3), we selected a subset of \(K\) features and prioritized our data collection and model development effort for these features. Currently, there are \(K=4\) features in this subset, which can easily be extended in the future. **Dataset Creation**. As described in Section 2.3, the data visualization method inspired us to develop ML models that can perform similarity detection tasks through \"viewing\" such heatmap images created for individual features. Note that it is not necessary for ML models to view the blue-yellow color-coded images as shown in Section 2.3. Although heatmaps are shown as color-coded images in this paper, in the context of ML models, they are stored as numerical matrices (32-bit float per cell) without any color information. Because of the shortage of music similarity data that are legally validated, we intentionally kept such treasured authentic data out of the dataset for training and validation, and used them for independent testing only. As part of this work, we created a training and validation dataset using \(N_{0}\) traditional songs that are copyright-free. Furthermore, we selected those songs, for which it was feasible to create their variants with available expertise and resources, and the similarity labeling was fairly certain when comparing songs and variants in the collection. For the work reported in this paper, \(N_{0}=6\), and we are adding more songs into the corpus for future work. The six songs are _Bella Ciao, Twinkle Twinkle Little Star_, _Row Row Row Your Boat_, _Ode to Joy_, _Happy Birthday_, and _Jingle Bells_, which are considered to be dissimilar. For every song in the corpus, we created some variations by using methods such as _transpose_, _ornaments added_, _chord altered_, and _swing genre_. With these variations, we expanded the song corpus from six to \(N=34\) music pieces. Each original song and its variations forms a song group, and any comparison between any two music pieces in the group is considered to be **similar**, while any comparison between two music pieces selected from two different song groups is considered to be **dissimilar**. These six songs and their variants are part of the dataset available on GitHub ([[https://github.com/asaner7/imagery-music-similarity](https://github.com/asaner7/imagery-music-similarity)]([https://github.com/asaner7/imagery-music-similarity](https://github.com/asaner7/imagery-music-similarity)), accessed on 30 May 2025). For each feature \(f\), we use the feature-based analysis and visualization methods described in Sections 2.2 and 2.3 to compare every pair of music pieces in the corpus, yielding \(N\times N=1156\) heatmaps. Note that these heatmaps include self-comparison  
Figure 2: Comparing the song “Under Pressure” with (**a**) itself, (**b**) “Ice Ice Baby” (similar), and (**c**) “Bitter Sweet Symphony” (dissimilar). These heatmaps were generated in the ensemble decision process (see Section 2.5), and each is a weighted sum of a set of feature similarity heatmaps with the ensemble weights. The temporal steps along the axes are of 1 bar song segments. heatmaps as well as both orderings of every pair of music pieces (i.e., A-B and B-A). The former feature essential patterns that an ML model is expected to recognize. The latter feature diagonally mirrored patterns that an ML model can benefit from being trained and validation-tested with both. These heatmaps are split into training and validation datasets at a ratio of 80%:20%. In order to balance the positively and negatively labeled data objects, we used a weighted random sampler with weights inversely proportional to the probabilities of class labels. **Training and Validation. ** With the comparative imagery as input data, we utilized convolutional neural networks (CNNs) [19] as the technical method for training music similarity detection models.