Bringing songs to life through music visualization | by Sejal Dua | Towards Data Science

This past fall, when I was studying abroad in Copenhagen, I met this designer who was working on a lot of cool visualization projects using Javascript. After a few conversations, I was completely sucked into his creative world. I was mesmerized by his Instagram posts sharing music visualizations in progress, and I ended up going down a rabbit hole that led me to discover lots of other creative geniuses doing similar work. I made a note to myself that I would circle back to this source of inspiration one day and get my hands dirty with a project of my own.
Before we get into it, let me just set the record straight that I struggle in the creativity apartment. I do not have a wild imagination — my artistic taste is pretty much limited to geometric designs that are satisfying to look at and maybe borderline hypnotic. For this reason, figuring out where to start was, without a doubt, the most difficult aspect of this project.
Oddly enough, my main inspiration for this project came from the TV show This Is Us. It is admittedly quite an emotional drama — please don’t judge me for watching it. However, one thing I am not ashamed to admit is that I am a huge fan of Siddhartha Khosla’s original score for the show. I think he is a brilliant composer, and sometimes I play his music on repeat at night to lull me right to sleep. I wanted to explore those tracks further and see if I could create a visual component that could be equally as soothing as the audio.
The first step was to get acquainted with the p5 Javascript library. p5.js is a library that is specially tailored for artists, designers, educators, and beginners with the purpose of making Javascript more accessible for coders looking to implement different kinds of applications.
I perused through a multitude of examples to figure out how to process audio. First, I played around with a sound frequency spectrum to see a basic example of audio being mapped to some type of visual. Then, I learned of a function called getEnergy() which returns the amount of energy (volume) at a specific frequency, or the average amount of energy between two frequencies. Moreover, predefined frequency ranges (“bass”, “lowMid”, “mid”, “highMid”, and “treble”) can be passed into the function. The only stipulation is that the analyze() function must be called initially because the getEnergy() function relies on the analyzed frequency data from that function call.
The frequency extraction code for my project was quite simple:
var fft = new p5.FFT();fft.analyze();var bass = fft.getEnergy("bass");var treble = fft.getEnergy("treble");var mid = fft.getEnergy("mid");
It was mapping the audio to the visual components that took a lot of trial and error with angles and math. I ended up using mouse positioning to dictate the number of line segments and radius for the polygons in the visualization.
pieces = map(mouseX, 0, width, 4, 8);radius = map(mouseY, 0, height, windowHeight / 4, windowHeight/2);
Then, I made some mapping and scale variables for the line segments.
var map_mid = map(mid, 0, 255, -radius, radius);var scale_mid = map(mid, 0, 255, 1, 1.5);var map_treble = map(treble, 0, 255, -radius, radius);var scale_treble = map(treble, 0, 255, 1, 1.5);var map_bass = map(bass, 0, 255, -100, 800);var scale_bass = map(bass, 0, 255, 0, 0.8);
Finally, I wrote a for loop to define the bass, mid, and treble polygon rings that would be rotating to the frequency of the audio throughout the song.
The result of this code can be seen in the video below.
One of the main reasons I chose to use the song “Jack in AA” from the This Is Us soundtrack was because I happened to have its MP3 locally on my computer and could easily upload it into the visualization. I wanted to be able to visualize other songs, so I implemented a pre-loaded track feature that allows you to click the number keys to switch the track being played. The additional track options (excluding “Jack in AA” that is preloaded as track number 0) are:
A R I Z O N A - Where I Wanna BeBleachers - Wild HeartDesiigner - PandaA.R. Rahman - Dil Se ReKhalid - VertigoShakey Graves - Family and GenusMumford & Sons - I Will WaitTravis Scott - HIGHEST IN THE ROOMGlass Animals, Denzel Curry - Tokyo Drifting
A R I Z O N A - Where I Wanna Be
Bleachers - Wild Heart
Desiigner - Panda
A.R. Rahman - Dil Se Re
Khalid - Vertigo
Shakey Graves - Family and Genus
Mumford & Sons - I Will Wait
Travis Scott - HIGHEST IN THE ROOM
Glass Animals, Denzel Curry - Tokyo Drifting
A good variety of genres is represented by these tracks. It was actually quite a challenge acquiring MP3 files without having to visit sketchy, virus-laden, and probably illegal sites. That being said, there is an option to upload your own track if you have a .mp3, .wav, or .ogg file.
The coloring of the visualization is quite straightforward. An array of four colors is used to make up the color palette. The four colors are given in the following order: [(background), (bass), (mid), (treble)].
To allow for color switching, I came up with a total of 5 color palettes (red, orange, green, blue, and purple) and then added a simple keypress listener to iterate through the color palettes. Pressing the “Enter” key triggers this functionality, as well as double clicking.
You can pause or play the track by clicking the “Space” key or by clicking the toggle button in the bottom left corner.
In the demonstration above, we see the song “Where I Wanna Be” by A R I Z O N A brought to life in the visualization. This song has a characteristically fast tempo, which is reflected by the frequent contractions of the polygons whenever the frequencies change in accordance with the beat. It is also very satisfying to hit the “Enter” key on each beat because the color changes make the visualization feel very vibrant, almost like you might find at a disco or nightclub.
The song “Tokyo Drifting” by the Glass Animals and Denzel Curry has been a favorite of mine as of late. My good friend sent it to me a while back, and I didn’t get what the hype was all about, but each time I listen to it, I come to appreciate it more. I recorded a snippet of the visualization in the video embed above primarily because the intro to the song is fascinating to watch. Since there are not a lot of audio channels being mixed at once during the first few seconds of the song, each semi-creepy sound has a direct correlation to some visual component. Then, at 0:27, there is a huge beat drop that completely sucks you into the rest of the track. The beat is quite prominent throughout the whole song, and its significance is reflected in the size of each beat pulsation represented by the light blue color in the video.
Automatically pull new music from some music streaming service so that I (and others) can visit this visualization app to listen to and experience new music.
Develop a more creative music visualization that is more thematic according to a song or artist rather than geometric.
Make a mobile app that runs this code in a more elegant way (some of the components are a bit janky in this web app).
Build a GAN that generates music continuously.
The source code can be found here:
github.com
This was just my first project involving music visualization using Javascript. I definitely intend to keep improving this project, and I also hope to pivot to other interesting applications and development tools related to music. If you would like to send me any feedback or inspiration, please contact me at sejaldua@gmail.com or through my website.
Thank you for reading!