# A Bayesian In-Play Prediction Model for Association Football Outcomes
## 4 Results
### Out-of-Sample Performance  
#### 4.4.1 Rank Probability Score (RPS)  
In order to measure out-of-sample predictive accuracy, we compare our model against other models. Specifically, we use the the ranked probability score for Win/Draw/Loss outcomes. The Brier score (BS) and the ranked probability score (RPS) are widely used measures to describe the quality of categorical probabilistic forecasts. The BS can be regarded as the special case of an RPS with two forecast categories [28]. The RPS is particularly appropriate for evaluating probability forecasts of ordered variables [29; 30] explained that RPS was the most rational scoring rule of those that have been proposed and used for football outcomes. For a single forecast the RPS is defined as  
\[RPS=\frac{1}{s-1}\sum_{i=1}^{s-1}(\sum_{j=1}^{i}(p_{j}-e_{j}))^{2} \tag{24}\]  
where \(s\) is the number of potential outcomes, and \(p_{j}\) and \(e_{j}\) are the forecasts and observed outcomes at position \(j\). A lower score indicates a more accurate forecast (lower error).  
We compare the RPS of our model with two other models, a pre-match forecasting model and an in-play forecasting model. Comparing with pre-match forecasting model, we might truly see whether in-match events contain additional information. A player based model proposed by [19], which is one of the state of the art model, is chosen as a comparator pre-match forecasting model. Their basic model for the scoreline in a football match is a bivariate Weibull count model described by [12]. In addition, the dynamic nature of team strengths are also incorporated into the player-based model. For in-play forecasting comparator model, our basic model is chosen, that is the pure birth process model.  
In order to compare with the RPS results given in [19], the same test set, one and a half season's data (570 matches) starting from 2014-2015 season to 2015-2016 season is chosen. The best result of the player-based model is 0.2020, and Table 2 presents the RPS values of our model and the pure birth process model, which prediction probabilities are on condition of scores at time \(T\). At kick-off, actually our model is the pure birth process model in that there is no in-match information to update teams' strengths. When information just from kick-off to 5 min is observed, the RPS values of our model and the pure birth process model are higher than that of the player-based model; the RPS value of our  
Figure 2: Box-plot of posterior predictive check for the goals’ difference against the replicated goals’ difference. In addition, we add the 95% and 80% confidence limits to the box-plot. The yellow-filled circles represent 95% confidence limits; the green-filled stars present 80% confidence limits.  
model is lower than that of the pure birth process model. When information of the first 10 min after kick-off is obtained, the value of the Bayesian model is lower than that of the player-based model, however, the value of the pure birth process model is still higher than that of the player-based model. From 20 min after kick-off, both in-match models perform better than the pre-match model. We can see that as more and more information is observed, the RPS values are getting smaller and smaller for the period of prediction getting shorter. So the results of later part of the match can not explain whether in-match events contain additional information. However, the results of first 20 min may illustrate that applying in-match information to calibrate strengths is useful.