CLASSIFYING SPOTIFY MUSIC PREFERENCES: A COMPARISON OF ALGORITHMS
By: Safya Osman
Tags: Python, Classification Algorithms, Machine Learning, Spotify, Scikit Learn

The second classification algorithm used is binary logistic regression with stochastic gradient descent. In logistic regression, we model the relationship between two qualitative variables using a logistic model to perform classification between two classes (0 and 1). We define:

where x^i is an augmented feature vector. Unlike K-nearest neighbors, the logistic model outputs the probability that a test example belongs to class 1. For a feature vector xi, we call its predicted probability f(xi) and define it to be:

σ(u) is our logistic function used for converting a scalar into a probability. We hope f(xi) is consistent with the ground truth probability yi for i = 1, 2,…, N. In order to do so, we must find β for which

is smallest. (We use the cross-entropy loss function to measure the difference between our predicted probability and our ground truth probability.)

In this case, we minimize L(β) using stochastic gradient descent, which incrementally moves β towards the direction of steepest descent (the opposite direction of the gradient of L(β)). Stochastic gradient descent starts by initializing a value for β called β0 . Then, for every sweep through the training dataset, or epoch, we randomly select an i at random and update β to be:

for t = 1, 2, 3,… up to however many epochs we’d like to perform. Here, α is our learning rate and the gradient of Li(β) is just the gradient of l(yi,f(xi)). Using ∇Li(β) instead of ∇L(β) is less taxing on our algorithm compared to normal gradient descent, and introduces randomness to the model. This helps to prevent overfitting since it inhibits the model’s ability too follow the noise in our training data too closely.

When we have reached our final β value, we can now classify our test data as follows: for each test example xi, if our predicted probability f(xi) ≥ 12 then we classify it as class 1, otherwise we classify xi to class 0.

In the code, I use scikit-learn’s implementation for logistic regression using both normal and stochastic gradient descent, as well as my own implementation from scratch. β is initialized to be all zeros, alpha to be 0.001, and the number of epochs to be 150. The code also computes the cost function at every iteration, and plots the resulting values. To compare differences in accuracy over the number of epochs, I test the algorithm using anywhere from 50 to 500 epochs, and plot each accuracy score versus the number of iterations.

In my comparison of the accuracy versus number of iterations of SGD, you can see how more iterations caused lower accuracy in Figure 1 below. The decrease in accuracy is a sign of overfitting — with more iterations, our model was more prone to following errors, thus creating a fit only desirable for the training data and not the test data. To curb this, the model stops after 150 iterations, since it seems to be both the point of convergence in Figure 2, and a high accuracy point in Figure 1.

Figure 1. Observing Overfitting Through Accuracy

Figure 2. Convergence of the Cost Function Value with SGD

Unexpectedly, both scikit-learn’s implementation and my own did not beat the score of K-nearest neighbors. My SGD implementation also received a higher score than scikit-learn’s implementation (😊). Sci-kit learn’s regular logistic regression algorithm yielded 53.8% accuracy, their SGD algorithm produced 51.2% accuracy, and mine got 58.5% accuracy.