Data in Politics — the Towns Fund and the Pork Barrel (Part 2)
Tags: Government, Data Journalism, UK Politics, Data Science, Machine Learning

After joining all of the above data together into a single table, we get something like the following:

2019 IoD & election results vs. Towns Fund decisions. Full table available here.

For each constituency, we have our seven deprivation scores, the one-hot encoded election winner (a column for each party, with a 1 if that party won the 2019 election and a 0 otherwise), the victory margin in points and our target column, “funded”, which is 0 if the constituency was chosen for the Towns Fund and 1 otherwise. The constituency name and code aren’t useful — we just leave them in the table for the reader’s benefit, and in case you want to do anything else with the data yourself (if you do, please let me know in the comments!).

Now that we have data, what are we gonna do with it?

The modelling

To achieve our goal, we will train a machine learning model to use the features we defined to predict the funding outcome. This is a simple classification problem where our classes are no (0) and yes (1) funding decisions, however there is a twist.

Normally in machine learning the goal is to make predictions for new, unseen data that will be presented to us in the future. We can teach our model how to make these predictions by providing a historic sample of data X (called the training dataset) along with some known outcomes Y. This approach is called supervised learning. The model calculates a function that converts the inputs (our political and deprivation statistics) into the desired output (the yes/no funding decision) with the smallest amount of error (ε) possible based on the correlations it can find in the dataset:

This equation is only half of the picture though. The training data is only one snapshot of all the possibilities that your model might have to deal with in the future. You will almost certainly come across new examples in the future that are a little different from all of the examples in the training data. Therefore if your model perfectly tunes itself to your training data, it becomes over-fitted on that one single snapshot and will probably perform poorly on future data samples.

The amount of mistakes a model makes on the training dataset is called the training error. This is easy for us to measure since we know what the answers (Y) were supposed to be for the training data. We don’t know what the answers are for future data though (otherwise there would be no point in building a model!) so the amount of error on this data, known as the generalisation error, is unknown. Generally we try to estimate the generalisation error by having another dataset, known as the test dataset, which is used to measure the performance of the model but never to train it. Model validation is a big subject that we won’t cover here, but if you’re interested in learning more this post provides a good introduction to the subject.

Our problem is slightly different though. There are no new towns for our model to classify; we are simply trying to recreate decisions that were already made and then study the model to find out how those decisions could have been made (this is called model introspection). Therefore we couldn’t give two hoots about the generalisation error; all we have is a single training dataset that we want to recreate the known outcomes for. This means we can indulge ourselves in all of the bad habits our data science professors beat out of us on day one. If this makes you uncomfortable, read no further!

We aren’t complete cowboys though. To meet our goal we must be careful to only break the rules of data science in specific and limited ways. We need to achieve two things: 1) recreate the observed funding decisions as accurately as possible (for which we will be ignoring the generalisation/testing error) and 2) inspect the behaviour of the model to understand how the decisions were made. For 2) we need to put our Good Data Scientist hats back on and start acting like grown-ups again.

Model introspection

To avoid drawing any false conclusions, we need to be careful about our choice of model and choice of features.

Model choice

There are many supervised learning approaches to choose from. We can categorise these approaches into linear models and tree-based models. Linear models are based on linear regression. Logistic regression (LR) and neural networks (NNs) are two examples of linear models that could be applied to our task. LR is probably not a good choice for us as it’s too simplistic. Its simplicity comes from making lots of assumptions about the data and its relationship to the target variable. NNs are the opposite — while they are flexible and don’t make too many assumptions about the data they are also very complex. This makes it difficult to inspect the model and understand how it works. Decision trees are a happy medium for us: flexible enough to model the outcomes well, and simple enough to easily inspect. We will use gradient-boosted decision trees from the XGBoost package in this case.

Feature choice

When inspecting a model, we have to beware of collinearity (or multicollinearity). This is where we have two or more features in our inputs that are strongly correlated with each other. Collinearity can affect the quality of predictions in linear models, but shouldn’t present such an issue in decision trees. However, even for decision trees collinearity can render any model introspection useless. When we are looking at the importance of a particular feature to the predictions of a model, if it is strongly correlated with another feature then the model could easily use that feature instead; worse still it could use one feature half the time and another feature the rest of the time. This could mean that our related features all end up with a low importance for the model, even though the latent feature (the unseen thing that links them all together) is actually very important.

Our deprivation statistics look particularly risky for this. If you have poor educational opportunities, that’s probably going to limit your future earning potential and employment opportunities. So there’s a good chance that the income deprivation index is correlated to education and employment deprivation scores. We should look at all of our features to see how strongly they are correlated to each other, and for every group of correlated features, just pick one; the one that does the best job of separating the “yes” funding decisions from the “no” ones. Here is a pairplot of all our feature correlations:

Feature correlations.

As expected, Income (INC) and Employment (EMP) deprivation scores are very strongly correlated. In fact, INC, EMP, EST (education, skills and training), HDD (Health deprivation and disability) and CRI (crime) deprivation are all highly correlated. Since HDD appears to have the largest difference between funded/non-funded constituencies, we will keep that feature and drop the other four. We will therefore define deprivation based on health/disability, barriers to housing and services (BHS) and living environment (LIV). All other forms of deprivation can be inferred reasonably well from these three measures alone.

The results

Now let’s train our model. As we mentioned earlier, we don’t care about generalisation error here so that makes things very easy as we’re happy to have a model that overfits. So we’ll just take a simple classifier and turn all the settings up to something ridiculously high so that we get 100% accuracy:

Our model overfits like crazy and perfectly replicates the actual funding decisions using the features we provided! Normally, this is a Very Bad Thing but in this case it’s exactly what we want. We can now look at our classifier model ( clf ) and see how it made these decisions. We will use two techniques for this. Let’s start with permutation importance.

Permutation importance is a simple concept: how well does the model make predictions if we take a certain feature away? If the performance doesn’t change much, then this feature can’t have been particularly useful. If performance is much worse, then that feature must have been important. And if the performance improves when we take a feature away (which gives a negative permutation importance score), then it actually has a detrimental effect on the model. If we care about generalisation error, we should consider removing such a feature.

Here are the permutation importance scores for each feature in our model:

Permutation importance for each feature.

This tells us that Health deprivation (which, remember, is strongly correlated to income, employment, education and crime deprivation too) is the most important feature. That’s reassuring. However, the next most important feature should play no part in the funding decisions: the election victory margin. Worse still, the next strongest predictor is whether or not the constituency returned a Conservative MP in the General election. We already knew from Part 1 that Conservative constituencies were more likely to get funding, we can also see now that marginality (constituencies that the Conservatives were fighting to hang on to, or fighting to win from another party in the upcoming election) also plays a part. Let’s dig into this further with another approach to analysing feature importance.

Shapley values are a way of analysing feature importance using game theory. Here we assume each feature is playing a game to “win” as much responsibility for each prediction as possible. The Shapley values (aka SHAP values) are a score assigned to each feature for each prediction. The higher the score for a feature for a given prediction, the more responsible the feature was for that particular outcome. This allows us to take a very detailed look at the relationship between features and outcomes. We can easily calculate Shapely values with the excellent Shap package.

Let’s start with a summary plot, which shows the average SHAP value across all predictions for each feature:

Average SHAP values per feature.

This tells a very similar story to the permutation importance values. However the power of SHAP is that it lets us see how these importances vary across different predictions. Let’s plot each prediction individually:

SHAP values per prediction.

A high SHAP value means that a feature made a funding decision much more likely to be a “yes”. A very low SHAP value means the feature strongly pushed the decision towards a “no”. A neutral SHAP value means the feature wasn’t very important for that particular prediction.

Each point on the above chart is coloured based on the feature value. So for HDD, if a point is red then this indicates a constituency with a high deprivation score. We can see that the more deprived an area is, the more that deprivation is factored into the decision, and it leans towards a “yes” which is what we’d expect. For the vote margin, it appears to be the opposite. This is bad. This means that a seat being marginal affects the decision, and the tighter the margin, the more it contributes to a “yes” decision. This is Pork Barrel Politics — allocating funding based on party political considerations at the expense of wider public interests. The winning party feature (CON) also tells the same story: the places that voted Conservative in the election (confusingly coloured red in the chart) are more likely to be judged favourably.

Let’s look at a few specific predictions to understand this more. Here’s Loughborough, the least deprived town on the Towns Fund list:

Loughborough (CON). Note that a final score of > 0.5 means a town gets funding.

Loughborough was won by the Conservatives in 2019 by 13 points. It is a bellweather constituency, meaning that it nearly always selects an MP from the party that goes on to win the election. Until the 2019 the seat was held by Nicky Morgan, a high-profile MP who stood down shortly before the election. The winning Conservative candidate was Jane Hunt, who was elected for the first time. The SHAP values show us that being a Conservative seat had a small effect on the final decision, but the marginality had a larger effect. HDD had a negative effect, meaning that the Town got funding despite its relatively low deprivation scores.

Halton (LAB).

Halton was the safest Labour seat that received funding. In fact, it’s one of Labour’s safest seats in the country, and they won by a 41 point margin in 2019. The SHAP values for Halton show us that political considerations hardly affected the outcome at all: it was almost all down to Halton’s high deprivation scores.

We can also look for any correlations between SHAP values and the feature itself:

SHAP values for Health and Disability Deprivation Index.

We can see in the above chart that the higher the deprivation score is, the larger its apparent effect on the funding decision. This is what we would hope to see. We can also see that the non-Conservative (which almost always means Labour) seats are more often at the higher end of the deprivation scale. This is another way of showing us what we discussed in Part 1: even though Conservative seats were more likely to get funding, they were also less likely to be deprived. Let’s look at the relationship for victory margin next:

SHAP values for 2019 victory margin.

This time we see a downward trend. The more hotly contested a seat is, the more this becomes a factor in the final funding decision.

Conclusion

In this post we used machine learning to learn more about the correlations between different measures of deprivation, political allegiances and the decision for whether or not a town was allocated funding. Just as in Part 1, we saw lots of evidence to suggest that the party political interests of the Conservative party played a significant role in the outcome.

Machine learning isn’t a smoking gun; it can’t tell us for certain what ministers and officials were thinking when they made their decisions. It can however tell us that correlations exist where they shouldn’t if the process was fair. A parliamentary watchdog committee conducted an inquiry that said the process gave “every appearance of having been politically motivated”. Our own observations would appear to support that conclusion.

The inquiry also criticised the lack of transparency around the qualitative criteria used in the process — the features that we had to omit from our analysis because no data has been published for them. The government response to the inquiry was:

We completely disagree with the committee’s criticism of the town fund selection process, which was comprehensive, robust and fair.

If they want us to believe that, they’re going to need to show us the data that proves it.