Predicting the 2020 Senate Races With the Cook Partisan Voting Index
By: Matt Thoburn
Tags: Politics And Elections, Regression, R, 2020 Senate Elections

ggplot(df,aes(x=Expected.Result.PVI ,y=Result)) + geom_point() + geom_smooth(method = “lm”, fill = NA)

Expected vs actual results and their regression line

ggplot(df,aes(x=Expected.Result.Incumbency ,y=Result,col=Incumbency)) + geom_point() + geom_smooth(method = “lm”, fill = NA)

Expected vs actual results for each incumbency class and their respective regression lines

ggplot(df,aes(x=X538.Expected.Result ,y=Result)) + geom_point() + geom_smooth(method = “lm”, fill = NA)

FiveThirtyEight expected vs actual results. Note how much less their points deviate from the regression line.

Results

As it will no doubt come as a shock to you, a simple regression model using a one or two variables trained on a manually assembled dataset is no match for the minds over at FiveThirtyEight and their model (This comparison was done against their Deluxe model, which combines polling, fundraising, past voting behavior, and more).

In order to quantitatively compare the models, we can perform another regression, this time comparing the expected results (based on the model in question) and the actual results. If we run these calculations for each model, we see that the PVI, PVI+Incumbency, and 538 models have $R²$ values of .85, .83 and .94 and RSEs of 8.5, 9.1, and 5.1 respectively.

It shouldn’t surprise us that theirs outperforms mine given that their model most likely includes all the information mine is capturing as well as additional information that mine isn’t privy to. It would be interesting to see how my models compare to all three of FiveThirtyEight’s models (They released three this cycle: Lite, Classic, and Deluxe, incorporating different levels of complexity. Lite uses just polling data, and Classic & Deluxe add additional variables on top of that). If I wanted a more even playing field pitting polls against PVI, it may have been beneficial to have pitted my models against their Lite model. But alas I was hoping for a compelling David vs Goliath narrative that didn’t quite pan out.

Conclusion

I think one of the important takeaways here is that its important to differentiate between a fundamentally sound model and one that happens to get lucky. I’m sure if I had spent enough time creating different models I would end up with one that happened to predict the winner of all the key senate races. I could have then taken to Twitter, tagged Nate Silver in a tweet and said “AHA! Your model gave democrats a 75 percent chance of taking the senate but my model said otherwise and it turns out they didn’t take the senate after all therefore my model is superior and I’m a statistical wizard!”. However said model might not actually hold up under scrutiny, and it might not generalize to future elections in the way a fundamentally robust model would. This would be a bad model.

It is important to note that I’m playing a little fast and loose with the notion of “right” and “wrong” here given that regression is inherently probabilistic. A regression model such as this doesn’t just predict a specific value, it predicts a distribution centered about some value, although ideally it seeks to minimize the variance of this distribution. So as long as the outcomes are within the reasonable bounds of our prediction intervals (i.e. results that should only happen 1% of the time not happening 50% of the time), we could say that our regression model is doing reasonably well, even if it isn’t hitting its target exactly on the bullseye every time.

While my models did seem to beat the conventional wisdom in a couple circumstances, they seemed to be wrong about as often as they were right (at least in terms of predicting a winner). In the case of my models, they seem to have identified a broad trend in behavior, but don’t capture enough information to be able to be used for particularly accurate predictions.

At the end of the day, it should come as a comfort to get some vindication that elections come down to more than just prior voting habits (PVI is essentially a weighted aggregate of past election trends for a given state or congressional district). Our regression analysis would indicate that it does in fact matter who campaigns choose to run and that how well they run their campaign does affect the outcome, and that elections are in fact not predetermined. The fact that Sen. Collins has been in the senate for almost as long as I’ve been alive despite Maine consistently voting for Democrats in that time is a testament to her skill as a politician and a campaigner (and I say that as someone who dislikes Sen. Collins immensely, politically speaking). She did face a closer election this time around than in 2014, but she didn’t seem particularly encumbered by the state PVI, nor did she seem encumbered by the fact that President Trump lost Maine by about ten points at the same time as she won by about seven.

Future ideas

One of the lingering questions I have is whether or not we could have improved upon our Incumbency regression with some more information. For instance Sen. Gardner only served one term and won his election by two points, whereas Sen. Collins has been in the senate since 1997 and won her last reelection by 37 points (in a state that voted for Obama by 16 points the presidential cycle before). Sen. McSally was appointed to fill John McCain’s recently vacated seat (by way of John Kyl’s resignation from the seat in 2018) despite having just lost a senate campaign for the other Arizona seat that year, and Sen. Jones won in an upset in a special election in Alabama against someone who may or may not have been a pedophile. Surely it would not be unreasonable to weigh these candidates differently, and perhaps there would be a way to better quantify the strength of a particular incumbent’s incumbency that would have resulted in better predictions.

Additionally, maybe there are other variables that could be captured to use in tandem with PVI and/or Incumbency to improve the model. I would think there would ideally be some way to capture the general level of enthusiasm around a candidate, as you would hope it would make a difference if parties ran candidates that people were actually interested and excited about as opposed to cookie-cutter partisan candidates. Some metric around small dollar donations and/or fundraising come to mind, although given the disappointing congressional results democrats had relative to their fundraising against republicans, I’m not inclined to believe that its a particularly fruitful line of investigation (and from what I understand, the political science literature isn’t convinced of it either). Perhaps there are social media or search engine metrics that could be captured to measure engagement online, although there are likely issues of sampling bias that would have to be addressed for that to be viable (See this study by Pew Research on the shortcomings of translating the Twitterverse to reality).

Postface

If you’ve made it this far, congratulations and thank you for bearing with me. What did you think? Did you love it? Did you hate it? Do you want to hire me to run your next campaign for office? Let me know in the comments so I can learn and grow.