The Devil’s Music
By: Callum Ballard
Tags: Music By Numbers, Python, Culture, Music, Data Science

At first glance…

Pitchfork has been one of the most prolific music publications of the last twenty years. Only two others have more reviews in the dataset, and since 2018, Pitchfork has published more reviews on a monthly basis than anyone else, aside from AllMusic.

The trends here are interesting in and of themselves — what causes this jump around 2011, and the fall in the following years? This is something we will investigate in a future blog. Note — we use a rolling average here since album releases are highly seasonal, with very few albums being released in December.

There is a very crude way to measure a publication’s harshness — take an average of each publication’s review scores, and rank them. Doing this, we see that there are many harsher critics than Pitchfork, which just scrapes into the top (or bottom?) thirty publications.

Of course, this analysis is flawed. We’ve already seen that the publications have reviewed different numbers of albums — maybe it’s the case that Pitchfork has simply reviewed more good albums, given the breadth of its catalogue. We thus need to introduce a more robust measure of harshness.

Difference to Metascore

We’ve already suggested that the Metascore is probably the best measure we have of an album’s true quality. Thus, we can identify a harsh review by comparing the Metascore with the individual review score.

Let’s isolate Pitchfork’s reviews. We can plot each review’s Pitchfork score against the corresponding Metascore. Albums that Pitchfork have reviewed ‘harshly’ therefore fall below the purple ‘x=y’ line. We can see a lot of points fall below that line — more than two thirds, in fact.

67% of Pitchfork’s scores are below the Metarating. 28% are above, and 5% are exactly the same.

We can boil this ‘harshness’ concept down into a single KPI — a review’s ‘Difference to Metascore’. This is easily calculated as the difference between the two numbers. In the chart above, it would be given as the vertical distance between the point and the purple line.

So for Pitchfork’s review of the latest Foals album, the Difference to Metascore would be 60 minus 78, i.e. -18 (seemingly quite harsh!)

The album’s metacritic summary, with Metascore

The same album’s Pitchfork review…

Therefore, the more negative ‘Difference to Metascores’ a publication has, the more harsh we can judge it to be. If we plot a histogram of Pitchfork’s score differences, we see that the majority of them are negative (which we knew from the above scatter plot).

On average, Pitchfork scores almost a whole half-star lower than we would expect, given the Metascores of the albums that it has reviewed.

If you’re wondering which albums are to be found in that long tail on the left, then we can only hope that you’re not a fan of Northern State…

Comparison across publications

One of the advantages of the ‘Difference to Metacritic’ KPI is that we can compare fairly across publications.

If we plot the mean score difference to Metascore for each publication, we actually see that Pitchfork doesn’t look especially remarkable — especially relative to other publications with a similar number of published reviews.

Many of the publications at the extremities of harshness and generosity (i.e. on the far left and far right of the above scatter) are much smaller concerns. The three harshest (Drawer B, Select, and Vulture) only published 318 reviews between them. The most generous are often ‘record of the week’ style columns, from which we would expect high scores.

If we isolate the top thirty publications (each with over 2,000 reviews in the dataset) we see that only a sixth of them review more generously than Metascores suggest they ought to — the only especially generous critic amongst them is Entertainment Weekly (make of that what you will). At any rate, by this measure, Pitchfork’s review scores don’t look particularly notable.

It’s the kurtosis, stupid…

However, something interesting emerges if we go beyond the average score differences for each publication, and instead look at distribution of these differences. A histogram works if we’re doing this for one publication (as we did for Pitchfork a few paragraphs ago), however to compare multiple distributions side by side, we can use Seaborn’s violinplot function. Below, we plot the distributions for the ten most prolific publications in the dataset.