# Stochastic Extensions of the Elo Rating System
## 2 Preliminaries
### Accuracy Metrics for Elo Rating Systems  
In order to determine the predictive accuracy of a rating system, we use several tools. The first one, provided by [PERSON] [1] (Chapter 2.6), is a statistical test of normality for a sample in which the players have preexisting ratings \(R\) and each player plays \(m\) games.  
Originally the sample was a chess tournament, or a set of tournaments with the same \(m\)[1] (Chapter 2.7).  
[PERSON] proposes to represent the values of the residues \(\varepsilon_{j}(R)\) in a histogram and to compare them with the frequencies of a normal variable with mean zero and the sample variance. Since \(\varepsilon_{j}(R)=\sum_{k\in M}\mathbb{E}[p_{j,k}|R]\) is the sum of \(m\) variables, if \(m\) is high enough and the rating differences are not too high, by the CLT, each \(\varepsilon_{j}\) should approximately follow a normal distribution, with variance \(\sum_{k\in M}\text{Var}(p_{j,k}|R)\leq\frac{1}{4}m\). More sophisticated normality tests, like the normal probability plot, can be carried out for the same variables.  
In order to compare the accuracy of different systems in the same sample, we propose looking at the average of the squared residues across all matches, i.e., the mean squared error of \(p\):  
\[MSE:=\frac{1}{|M|}\sum_{k=1}^{|M|}\bigl{(}p_{a_{k},k}-E[p_{a_{k},k}|R]\bigr{)}^ {2} \tag{14}\]  
We can decompose this sum using the known expression of the mean square error, which is the square of the bias plus the mean variance:  
\[E[(X-a)^{2}]=(E[X]-a)^{2}+\text{Var}(X) \tag{15}\]  
In particular, when applied to the results of the games, we obtain:  
\[MSE=\frac{1}{|M|}\sum_{k=1}^{|M|}\bigl{(}E[p_{a_{k},k}]-E[p_{a_{k},k}|R]\bigr{)} ^{2}+\frac{1}{|M|}\sum_{k=1}^{|M|}\text{Var}(p_{a_{k},k}) \tag{16}\]  
The advantage of this measure is that we do not require any of the assumptions of a normality test. The sample can be extended in time (we can update the ratings between games using Equation (1)), and we do not need every player to play the same number of games. Also, if we apply two different rating systems to the same number of games, we can look at the variance reduction and extrapolate p-values to determine if one system is better than another.  
However, unlike in a linear regression model, the results \(p_{k}^{a}\) do not have a constant variance, so the expression \(MRV:=\frac{1}{|M|}\sum_{k=1}^{|M|}\text{Var}(p_{a_{k},k})\) (mean result variance) can only be understood as the expected variance of a game, i.e., \(\mathbb{E}[\text{Var}(p_{k}^{a}|r_{a_{k}}-r_{b_{k}})]\), dependent on some distribution of the rating differences \(r_{a_{k}}-r_{b_{k}}\). We can only compare results from different samples if we assume this average variance to be similar, but this is not the case in general (for instance, \(MRV\) decreases as the dispersion of the ratings increases).  
If we assume the results are binary (\(p_{k}^{a}\in\{0,1\}\)), we can use other approaches. For instance, if \(F_{X}\) is logistic, as we saw in Equation (10), the model is equivalent to that of a logistic regression. This allows us to compute statistics like the deviance \(D\), which is approximately distributed as a chi-squared variable for large samples, by [PERSON] theorem [12]:  
\[D=2\cdot\ln\biggl{(}\frac{\text{likelihood}_{Elo}}{\text{likelihood}_{ saturated}}\biggr{)}\xrightarrow[|M|]{}\chi_{n}^{2} \tag{17}\]  
where, as before, \(n\) is the number of players.  
Finally, the effectiveness of the model can be gauged more visually by plotting the receiver operating characteristic (ROC) curve. Again, this does not work with other possible results besides a win or a loss, such as a draw (\(p_{k}^{a}=\frac{1}{2}\)), unless the non-decisive results are removed or imputed (for instance, to zero and one with probabilities \(1-p_{k}^{a}\) and \(p_{k}^{a}\)).