MIDI Music Data Extraction using Music21 and Word2Vec on Kaggle | by Waldecir Faria | Towards Data Science

Kaggle is a community site for data scientists and machine learners which offers several features to support their work and research. One of those feature is an online editor of scripts and notebooks which are called of “Kernel” by them. Users can create and edit Kernels online to share knowledge about some topic like machine learning algorithms or Python data structures.
I was taking a look there to see if I could find anything related with music or MIDI files but I didn’t find anything. There are few databases and kernels and most of them are focused on analyzing music recommendation and preferences.
This way I decided to create a new kernel there to learn more about MIDI files and play with the Word2Vec over harmonic sequences. In this article I’ll describe in high level some results which I achieved. You can find all code on my Kaggle Kernel link.
Once I was reading about music generation using neural networks and remembered another article about Explainable Artificial Intelligence. Since neural networks have “hidden layers”, it becomes harder to understand why the machine “choose” to create the music using a specific structure.
Then I thought, instead of letting the machine do everything, we could use the computer just to help music students to understand better musical compositions. Of course, a lot of people thought the same thing. For example, on Youtube you can find several program videos that shows which chords are being played at the music.
My idea was to create a Kernel where you could choose a set of MIDI musics, analyze them and extract knowledge focused on those compositions. This way I could answer questions like “which chord Beethoven would use here?” or “which Key Signatures are more used on Sega Genesis Sonic games?”
Usually when we are talking about music files, we think on files extensions like MP3 extension, which is an audio coding format for digital audio. Since they represent the music’s audio final format to be listened, concepts like “notes” or “chords” became hard to visualize.
Another way of representing a music is a MIDI file. They are a very compact way of representing a sequence of notes being used on different instruments. Nowadays it is pretty easy to download a MIDI file from your favorite artist and start to play with it, transforming a piano solo into a guitar one or isolating that bass section which you love.
Usually a MIDI file has less than 20kb while a MP3 one can have more than 4MBs according to the compression configurations. Moreover, MIDI files describes all notes from a composition, so we can check easily the music structure using any MIDI editor. Finally, after extracting any relevant information, we can export a common audio file from the MIDI file and apply other audio processing techniques on it, combining the better from both worlds.
There are several libraries to manipulate MIDI files programmatically. Music21 is one of them. Quoting the definition from their site:
Music 21 is a Python-based toolkit for computer-aided musicology.
People use music21 to answer questions from musicology using computers, to study large datasets of music, to generate musical examples, to teach fundamentals of music theory, to edit musical notation, study music and the brain, and to compose music (both algorithmically and directly).
I choose to use it in this project because:
It uses Python code
It looks pretty robust and there are people working on its code since 2006 until now
It supports MIDI files well
It has music analysis methods based on note sequences
It can draw plots and music sheets easily
Music21 makes easy to extract notes from a MIDI file. You can get all notes from each instrument and work over them as you wish. I used a Sonic’s Green Hill MIDI file for the next pictures:
This library offer lots of methods to handle note sequences, in this project I mainly used a method to analyze key signatures and a method to get a chord from a note combination. For example, a note sequence [C, E, G] would result the First chord from the C-major scale.
A single music can have hundreds of notes which makes it harder to analyze it. We can simplify it based on its harmony to make it easier to understand while losing some details like the melody. From Wikipedia:
In music, a reduction is an arrangement or transcription of an existing score or composition in which complexity is lessened to make analysis, performance, or practice easier or clearer; the number of parts may be reduced or rhythm may be simplified, such as through the use of block chords.
Manually there are several ways of doing that, considering the context on the measure being analyzed, the music’s current key signature and the set of notes on the current measure. In this project I followed these steps:
Find the music key signature
For each measure, sum the time which each note is pressed and choose the 4 most frequent ones
Use Music21 to find the Chord and its function based on the key signature
Simplify Chord names to avoid names like “bII#86#6#5”, even if it means to lose chord information
Consider Bach’s Prelude in C-Major, I was able to get the following sequence using my method on it:
['I', 'ii42', 'vii53', 'I', 'vi6', 'ii42', 'V6', 'i42', ...]
My method isn’t perfect, guessing several wrong chords, but now we can describe a Bach’s Prelude harmony using a string for each measure. Since that Prelude has 32 measures, in the end we get a list of 32 words describing its harmonic progression.
If we could get all Bach’s works and do the same, we would end with a sequence of harmonic sequences which we could analyze and find some interesting relationships like which chord Bach would replace by another one in a composition.
Word2Vec is a technique to do that with texts, citing a quote from this article:
The idea behind Word2Vec is pretty simple. We’re making an assumption that the meaning of a word can be inferred by the company it keeps. This is analogous to the saying, “show me your friends, and I’ll tell who you are.”
If you have two words that have very similar neighbors (meaning: the context in which its used is about the same), then these words are probably quite similar in meaning or are at least related. For example, the words shocked, appalled, and astonished are usually used in a similar context.
In my Kaggle Kernel I downloaded 450 Sonic MIDI files from Sega Genesis games and generated a harmonic reduction sequence for each file. Then I trained a word2vec model to extract more information about the chord use there.
With the trained model I was able to answer several questions based only on Sonic compositions, some examples:
Finally, I created a method to compare several musics using my model. The word2vec trained model transforms each harmonic reduction sequence into a numeric vector and we compare them using cosine similarity. I chose this Green Hill MIDI file as the base and compared with all available versions of:
Emerald Hill Theme (Sonic 2) — most similar version got score 0.990
Hydrocity Theme (Sonic 3) — most similar version got score 0.991
Sandopolis (Sonic & Knuckles) — most similar version got score 0.963
Then we could say that Sandopolis theme has the most different harmonic structure when compared with Green Hill theme.
I hope that you enjoyed this article and learned more about music theory and Sonic themes.
Regarding word2vec use, I need to find a good way to measure the model’s accuracy but the tests which I demonstrated previously based on music theory presented nice results. There are also a lot of space to improve on the other methods and I have several ideas to play more with that data when I have more free time. Please comment if you have questions or feedback. See you!