Rating Sports Teams — Elo vs. Win-Loss | by Blake Atkinson | Towards Data Science

There are many ways to determine who is the best team or player in any sport. You can look at the last 5 games. The last 10 games. You can use score differential. You can rate them on which teams “feel” the best. You can look at the roster and see which team has Tom Brady or LeBron James. Whether it be power rankings, divisional rankings, or anything else — there are scientific ways to rate teams and there are unscientific ways. The following is an attempt to scientifically decide on a good ratings system.
First of all, I don’t think I can beat 538’s thorough description of Elo. The Wikipedia page is also packed full of good information. I will however give a quick synopsis.
The Elo rating system is a very simple yet extremely effective ratings system. It was originally developed for Chess, but nothing about it is Chess-specific. Elo, in its original form, can be used for any head to head game. Arm wrestling, Scrabble, Pong, you name it. One can also treat teams, such as basketball teams, as “players” facing off against each other.
Each player or team has their own Elo rating. It is convention that new players start with a rating of 1500, but 1500 is somewhat arbitrary. When two players play, they automatically each “wager” some of their Elo points before the game. The size of this wager is known as the K-value. The winner is awarded points after the match and the loser loses points after the match. The amount of points awarded depends on the relative difference in initial ratings. For example, if a 1700 skill player beats a 1300 skill player, not many points are awarded — this outcome was expected. On the other hand, if the 1300 level player beats the 1700 level player, almost the entire pool of wagered points is awarded to the 1300 level player and deducted from the 1700 level player.
The K-value, or amount of points wagered, depends on the game. In a sport such as baseball, the K-value would be small. Even the best baseball teams struggle to win more than 70% of their games. Also, they play 162 games in the MLB. So if the best team in the MLB loses three straight games it’s probably not a huge deal and they still might be the best team in the league. You don’t want a large K-value or else Elo ratings will overreact to the losing streak. However, in the NFL or college basketball, the best teams are expected to win more than 80% of their games. In the NFL, there’s a small sample size as well. It’s desirable in that case to allow more volatile Elo ratings — a team with a 3 game losing streak is probably not the best team in the league. Also, over a short season, you want the ratings to adjust quickly.
If we’re interested in predicting the outcome of games, Elo ratings are powerful predictive tools. They’re even more predictive than win-loss records and come close to prediction markets. That’s assuming you have a good K-value. Once you are somewhat familiar with Elo ratings, you can guess good K-values. The purpose of this post is to give you a scientific way to arrive to the best one.
I claim that Elo scores are better than win-loss records in describing the skill of a player or team. Let’s not take that for granted. First, to demonstrate the point, I will design an abstract game in which some players are better than others, have them play each other. Players will have a win-loss record and an Elo rating. We will then be able to compare the ratings in a friendly environment to compare them. In a future post, I will use easy-to-obtain sports data and see if the process works in the real world, and not just a friendly experiment. The code for this experiment is available on my Github page.
In the real-world, there are messy complications that Elo can’t take into account. For example, a team will draft players in the offseason and become better or worse. If a star player is traded or injured, Elo has no idea. So let’s design a game that doesn’t have messy complications.
There’s a new game called Sportball, and a major league with all the best Sportball players just formed. Fans from around the globe travel to see their favorite Sportball teams. Major League Sportball has 32 players, and players always play at the midpoint between the cities that they are from (to neutralize home field advantage). While ticket offices don’t like this move, people who calculate Elo ratings love it.
To simulate true ability, there will be a “true rating” for each player. When two Sportball players play, a random variation will be added to the true rating so that lesser teams can upset better teams. Otherwise, it would be a boring experiment. The random variation is basically a way to simulate luck and how players are feeling that day.
Lastly, to simulate change over time, each player’s true ratings will randomly be nudged higher or lower after each game. Season win-loss records will not be able to adjust to these nudges yet Elo will be able to do so. This gives Elo an advantage that it would also have in the real world. At the end of the 1000 game season, we will compare true rating, win-loss record, and Elo scores.
Let’s first create a player class to track all these attributes.
class Player(object):    """Player Object"""    def __init__(self, name, rtg, elo, nudge):        super(Player, self).__init__()        self.name = name        # true rating        self.rtg = rtg        # elo rating        self.elo = elo        # wins, losses, ties        self.wins = 0        self.losses = 0        self.ties = 0        self.games_played = 0        # how much to nudge ratings        self.nudge = nudge    def nudge_rating(self):        # decide to nudge up or down        direction = random.randint(0,1)        if direction == 0:            self.rtg += self.nudge        else:            self.rtg -= self.nudge    def add_win(self):        self.wins += 1    def add_loss(self):        self.losses += 1    def add_tie(self):        self.ties += 1    def played_game(self):        self.games_played += 1
I like to set up a settings.py file to easily adjust parameters. So let’s create some basic settings for our experiment.
season_settings = {    # the way it's coded, it must be even number of players    'num_players':32,    'num_games':1000,    # average ratings for the teams    'avg_rtg':100,    # standard deviation of ratings (higher -> more difference in player skill)    'std_dev': 5,    # game by game variation amount:    'game_var':7.5,    # after each game, how much does the true rating change?    'rtg_nudge':0.2}elo_settings = {    'init_elo': 1500,    'K': 2,    'beta':400}
I ran this a couple of times beforehand and decided on a reasonable K-value. The purpose of this article isn’t to explain Elo, rather to maximize how effective Elo is. I encourage you to copy my Github repo and play with these values yourself. If you want to know more about beta, and K-value I encourage you to read the links I gave above. Once we’ve decided on settings, we can create a league:
# array to store playersleague = []# table to track wins/lossestable_array = []# create leaguefor p in range(season_settings['num_players']):    # initialize players    player_name = fake.name()    # assign random initial true ratings    player_rtg = random.gauss(season_settings['avg_rtg'], season_settings['std_dev'])    player_elo = elo_settings['init_elo']    nudge = season_settings['rtg_nudge']    player_entry = [player_name, player_rtg, 0, 0, 0, player_elo]    table_array.append(player_entry)    new_player = Player(player_name, player_rtg, player_elo, nudge)    league.append(new_player)league_table = pd.DataFrame(table_array, columns=['Name', 'True Rating', 'Wins', 'Losses', 'Ties', 'Elo'])league_table = league_table.sort_values(by='True Rating', ascending=False)
Note: I’m using the faker package to create fake names. Now we have 32 players with a normal distribution of skill:
So it’s time to play the first season of Sportsball! I won’t give all the code, but I will show the for loop I’m using to run matchups:
for i in tqdm(range(season_settings['num_games'])):    # create random matchups (pops from league list)    matchups = create_matchups(league)    # reset league    league = []    # play games    for matchup in matchups:        p1 = matchup[0]        p2 = matchup[1]        p1_score, p2_score = play_game(p1, p2)        p1, p2 = update_players(p1, p2, p1_score, p2_score)        league.append(p1)        league.append(p2)
Note: I’m also using the tqdm package to show a progress bar. Again, I encourage you to go to my Github to see all the code. Here are the results! (Sportsball doesn’t have a playoff yet, as the managers couldn’t agree on the terms).
Congratulations to Erin Jones! She began the season with the best true rating, and ended the season with the best true rating. The most skilled player won the most! No offense to Erin, but we don’t really care. The more interesting case might be Brandon Nichols, who began the season 12th best and ended 2nd best. Both his Elo and his win count suggest he hasn’t been second best for all that long. Perhaps he improved drastically at the end of the season?
All of a sudden we have a dilemma. How do we know how good a ratings system is? We can take the correlation of the win/loss column (awarding half wins for ties), and the True Rating, to get an idea of how good using win-loss record is at reflecting True Rating. The correlation is 88.2%. However, this is imperfect. If Brandon Nichols improved drastically at the end of the season, win-loss record wouldn’t accurately reflect the True Rating (which clearly it doesn’t). Elo rating also clearly hasn’t caught up yet to Brandon’s true skill. It still thinks Jasmine Brown is second best. Ha! Should we increase the K value so that Elo adjusts faster? Hmmm...
And, if you’re curious: the correlation of Elo and True Rating is 88.6%. This is far from definitive proof that Elo is better than win-loss. In fact, when I run this code multiple times, the win-loss is sometimes correlated better. The high correlation does however suggest that Elo is a decent method of rating players.
So we need a method to rate a ratings system. It’s like ratings inception. Which brings me to a magical addition, Brier scores.
Brier scores are a straightforward way of scoring predictions. Like Elo, I’m not the one who can explain it best. They are a central component of one of my favorite books, Superforcasting. I will do my best to give a quick synopsis.
Let’s say I’m forecasting the weather. I say there’s a 30% chance of rain. Then it rains. It’s tempting, and very human, to say “well that’s an awful prediction”. Humans (and I’m guilty of this too) often assume if someone is on the wrong side of 50%, they are wrong. Whenever someone says there is a 65% chance of something, and it doesn’t happen, it feels like that person is wrong. But that person could’ve been exactly right. In fact, someone might suggest there is a 5% chance of rain, and then it rains, and that person is still right. I promise I’m not a weatherman or weatherman apologist. We simply live in an unpredictable world, and unlikely events occur. Luckily we are Sportsball analysts, and we get a 1,000 game season to figure out our error!
Brier scores basically sum up prediction error. A good forecaster will have a minimal Brier score. I’ll take the Wikipedia examples on a rain forecast to demonstrate:
If the forecast is 100% and it rains, then the Brier Score is 0, the best score achievable.
If the forecast is 100% and it does not rain, then the Brier Score is 1, the worst score achievable.
If the forecast is 70% and it rains, then the Brier Score is (0.70−1)^2 = 0.09.
If the forecast is 30% and it rains, then the Brier Score is (0.30−1)^2 = 0.49.
If the forecast is 50% then the Brier score is (0.50−1)^2 = (0.50−0)^2 = 0.25, regardless of whether it rains.
So, there is a tradeoff here. I can choose to be very conservative with my rain forecasts. I can say there’s a 50% chance of rain every day. In that case, no matter what happens, my Brier error is 0.25 per prediction. That’s completely useless. Any error above that is therefore bad, and in some cases, it might be better to take the opposite of the prediction.
The strength of Brier scores is that it penalizes overconfident forecasters. You’re rewarded for predicting “I’m sure of something” when the outcome is near certain, and also rewarded for predicting “It might happen, I’m not sure” when the outcome isn’t certain.
There are weaknesses to Brier scores, and I’ll elaborate more in future articles. Let’s say I’m in the desert, where it barely ever rains. Then I can have a reasonable Brier score by predicting 0% every day. On one hand, it feels like cheating. The one or two times it does rain I will have a huge error (1 each day). This is a severe penalty. On the other hand, there is some degree of knowledge going into the prediction (that I’m in a desert) and so a smaller Brier makes sense. Ideally, if I’m a decent forecaster, I’ll increase my chance of rain prediction before it rains...obviously...
Elo rating naturally gives a percent chance that a player will win a game. It’s part of the intrinsic calculation. Likewise, win-loss record can do the same. If a player or team wins 76% of their games, it seems reasonable to expect they will win a random game 76% of the time. Therefore, we can calculate Brier error on both. Since Elo rating takes into account opponent strength, and also should adjust over time, we would expect Elo to outperform win-loss record. But let’s see!
I think the easiest way to track error is to add Brier error and Win-Loss error as player class variables. After each game, the both the win-loss prediction and the Elo prediction for each player will be compared to the result and Brier error will be calculated. It will then be added to the season long error total for the player. At the end of the season, all the season long errors from all the players will be added together. We’ll then divide by number of games and number of players to get the error per game. Let the best ratings system win.
Wow! That’s unexpected. Win-Loss record is much better at predicting the outcome! The exact error values are 0.41/game for Elo and 0.27/game for the W-L record method. Compared to our 50/50 benchmark from earlier, that’s really bad! That means that we really need to calibrate Elo. Let’s try adjusting K-value to improve Elo.
It turns out the Elo ratings are overreacting to every game. I changed the K-value to 0.5 and the Beta to 400, and that caused the Elo ratings to routinely beat the win-loss record. All in all, the Elo error would average about 0.244 and the W-L record would average about 0.262.
I then started playing with the season length. In longer seasons, like 10,000 games, it was much better to have a very tiny K-value. It seemed that Elo would run away and become inaccurate in super long seasons. I then changed the season length to 10 games and 100 games, and Elo routinely beat 50/50 predictions and W-L record with high K-values (such as 5).
0.24 sounds bad, and it kind of is. However, if you’re perfectly predicting a 70% win probability, then your error is 0.21. If you’re perfectly predicting a 90% win probability, then your error is 0.09. Brier scores don’t do a great job of rating predictions that are naturally close to a toss up.
It turns out that there might be a better, more popular way to calculate prediction error. Stay tuned!
As you can see, for any game that’s actually a toss up, even with perfect prediction accuracy — your Brier score won’t be much better than 0.25. The perfect Brier score can be found with an obvious equation (except I had to graph it to see it): -x2 + x.
Brier scores are great at identifying bad predictions. However, when I’m simulating near toss-up games, they struggle because the best score possible approaches 0.25.
When game outcomes have less variation, i.e. a team can win 95% of the time, Win-Loss records seem to perform better than Elo.
When there is more true skill mobility (the rtg_nudge setting), Elo outperforms Win-Loss as expected.
Over super long seasons, Elo gets out of control. This isn’t necessarily a knock on Elo, it’s kind of a bug in the experiment — due to random walk mathematics, certain matchups have an actual win expectation above 100%.
In order to quickly get accuracy at the extremes (I.e, when the best team plays the worst team), you have to raise your K-value. Raising the K-value hurts the accuracy of the mid-tier teams. So there is a trade-off — either be accurate on the uneven matchups or be accurate on the toss-up games.
All in all, I feel like I’ve learned a lot.
Future posts and goals:
Improve Elo through various means. Margin of victory, keeping it stable, etc.
Test regressions that use both Win-Loss and Elo
Perhaps use Win-Loss differential as another baseline tool
Operate on real data