How We Built a Content-Based Filtering Recommender System For Music with Python
By: Sammy Lee
Tags: Recommender Systems, Postgresql, Django, Pandas, Scikit Learn

High Fidelity a movie that came out 19 years ago about a record store owner named Rob Gordon who’s employees are supposedly so knowledgeable about music that they actually stop customers from buying music they want to buy. I mention this because before we had Netflix and Amazon and YouTube, real human beings in the flesh were the closest to personalized recommender systems we had.

The record store owner who knows what you like and recommends the newest Blink-182 or Green Day album, the restaurant server who’s tasted everything on the menu and knows exactly what you want based on what you had before, or the random stranger on the street who tells you the fastest and easiest way to get to a place you’re looking for — these are all recommender systems in the flesh — and very effective.

The problem is they don’t scale. And couldn’t scale until the internet arrived with things like Google. And even then there was no way to effectively evaluate the recommendation process until the arrival of data science and its ability to deal with lots of data.

Recommender systems in general can be divided into two types:

Collaborative-Based Filtering: Serves recommendations based on User similarity — using kNN (k-Nearest Neighbor) or matrix-factorization algorithms. Collaborative Filtering is the gold standard of personalized recommender systems, but you need lots and lots of User data which is why apps like YouTube and Amazon are able to do it so effectively. Content-Based Filtering: Serves recommendations based on the meta-data or characteristics of the very thing you are trying to recommend. If you’re recommending things like movies, then you would use genre, actors, directors, length of movie, etc. as inputs to predict whether you’d like a movie or not.

For MoodiBeats we ended up going with Content-Based Filtering due to the limitations of our data.

Part II: A Glimpse into the beginnings of MoodiBeats

Going back to our Lambda Labs project there was some decent amount of struggle within our team in the planning stages of MoodiBeats. One of the major problems with trying to integrate machine learning into a web app that doesn’t exist yet is the chicken or the egg problem — how do you design a data science driven frontend without the actual data, and how do you get the data for a website whose specification you aren’t so sure about?

Initially the data scientists wanted ready-made CSV files to work with so we spent almost two weeks analyzing the last.fm dataset and the infamous FMA dataset. Eventually, wanting to avoid anything having to do with copyright issues and the impracticality of letting Users download songs the FrontEnd team decided on using YouTube’s API and Player for no copyright music only. This forced the data science team to completely scrap all the work done on the last.fm and FMA datasets and refocus on trying to grab data from the YouTube v3 API in the middle of the project.

Part III: Let’s Build a barebones Django backend as a REST API

Warning 1: I’m going to rapidly build a Django backend with minimal explanation so readers who don’t have too much experience with Django, but are interested can consult the countless tutorials here on Medium or YouTube or here.

Warning 2: Parts III and IV will necessarily be long. However, know that what we’re doing here is actually building out a machine learning pipeline that will:

Automatically retrieve data from the YouTube API v3

Run a machine learning model

Expose both YouTube data and the machine generated predictions (in this case moods) as RESTful endpoints

And therefore non-trivial and perhaps highly useful for data scientists who want to collect a huge quantity of novel data and have it in a form accessible to the rest of the world. *If you care only about the data science part go ahead and skip to Part V.

On the command line:

"""Working within a virtual environment is highly recommended. For this project either Conda or Pipenv is sufficient. I'm using Python 3.6 and Django 1.11.20 and PostgreSQL for this project.""" $ mkdir MoodiBeatsAPI && cd MoodiBeatsAPI # psycopg2 is for interfacing with PostgreSQL database

$ pip install Django==1.11.20 psycopg2 # don't forget the trailing period

$ django-admin startproject music_selector .

Now open the the project folder (MoodiBeatsAPI) in a text editor of your choice, lots of people use VS Code nowadays, I still use Sublime Text .

Django ships with SQLite3 as a database, but my preference is to always use PostgreSQL, so if you don’t already have PostgreSQL I suggest you install it on your system.

Your project structure should look exactly like this:

. ├── manage.py └── music_selector ├── __init__.py ├── settings.py ├── urls.py └── wsgi.py

First create your PostgreSQL database:

$ psql -d postgres postgres=# CREATE DATABASE moodibeats; # You can verify that the database has been created by running postgres=# \l # And exit

postgres=# \q

Go into your settings.py and make some changes:

### Change this: DATABASES = {

'default': {

'ENGINE': 'django.db.backends.sqlite3',

'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),

}

} ### To this: DATABASES = {

'default': {

'ENGINE': 'django.db.backends.postgresql_psycopg2',

'NAME': 'moodibeats', # Name of our database

'USER': 'sammylee', # This would be different for you

'PASSWORD': '',

'HOST': 'localhost',

'PORT': '5432', }

}

On the command line:

$ python manage.py migrate

$ python manage.py createsuperuser # Create your credentials $ python manage.py runserver

# Append 'admin' at the end of your localhost to gain access to your # Django admin app # Go to http://127.0.0.1:8000/ on your web browser# Append 'admin' at the end of your localhost to gain access to your # Django admin app

If everything went smoothly, you should see this: