Uncertainty-Aware Reinforcement Learning for Risk-Sensitive Player Evaluation in Sports Game
Authors: Guiliang Liu
Yudong Luo
Oliver Schulte
Pascal Poupart
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement Learning
Uncertainty Estimation
Sports Analytic
Agent Evaluation

Abstract: A major task of sports analytics is player evaluation. Previous methods commonly measured the impact of players' actions on desirable outcomes (e.g., goals or winning) without considering the risk induced by stochastic game dynamics.  In this paper, we design an uncertainty-aware Reinforcement Learning (RL) framework to learn a risk-sensitive player evaluation metric from stochastic game dynamics. To embed the risk of a player’s movements into the distribution of action-values, we model their 1) aleatoric uncertainty, which represents the intrinsic stochasticity in a sports game, and 2) epistemic uncertainty, which is due to a model's insufficient knowledge regarding Out-of-Distribution (OoD) samples. We demonstrate how a distributional Bellman operator and a feature-space density model can capture these uncertainties. Based on such uncertainty estimation, we propose a Risk-sensitive Game Impact Metric (RiGIM) that measures players' performance over a season by conditioning on a specific confidence level. Empirical evaluation, based on over 9M play-by-play ice hockey and soccer events, shows that RiGIM correlates highly with standard success measures and has a consistent risk sensitivity.

Uncertainty-Aware Reinforcement Learning for Risk-Sensitive Player Evaluation in Sports Game
Guiliang Liu Lin Yudong Luo 1.3 Oliver Schulte', Pascal Poupart 1.
University of Waterloo, "The Chinese University of Hong Kong, Shenzhen, Vector Institute, "Simon Fraser University
liuguiliangdehhk cn, 1uo.ung. oschulte@cs. sfu. ca, ct.ppoupartuuaaterloo..oo
Abstract
A major task of sports analytics is player evaluation. Previous methods commonly measured the impact of players' actions on desirable outcomes (e.g., goals winning) without considering the risk induced by stochastic game dynamics. In this paper, design uncertainty-awe aware Reinforcement Learning (RL) framework to learn a risk lisk-sensitive player evaluation metric from stochastic game dynamics. To embed the risk of player's 's movements into the distribution of action- action-values. we model their 1) aleatoric uncertainty, which represents the intrinsic stochasticity sports game, and 2) epistemic uncertainty, which is due model's insufficient knowledge regarding distributional Bellman operator and a feature-space density model can capture these uncertainties. Based on such uncertainty estimation, we propose a Risk-sensitive Game Impact Metric (RiGIM) that measures players' performance by conditioning on specific confidence level. Empirical evaluation, based on over 9M play-by-play ice hockey and soccer events, shows that RiGIM correlates highly with standard success measures and has consistent risk sensitivity.
(OoD) samples. We demonstrate how a
season
Introduction
The advancement of player tracking and object detection systems enables data-driven analytics for professional sports players. A common approach to evaluating the contribution of players is to quantify their action impacts. Previous performance metrics T 2 impact of an action on scoring or winning game. However, actions with significantly different distributions of impact can have the same expectations. As a result, the expectation- based metrics cannot differentiate the risk-sceking actions from the risk-averse ones. How to distinguish these actions and assign proper credits to the players remains a fundamental challenge in sports analytics.
computed the expected
An important step toward risk-sensitive evaluation metrie is model the distributions of action values. To achieve this goal, distributional Reinforcement Learning (RL) [5] predicts the supporting quantiles of action-value distributions. Previous distributional RL methods [6] 7 g [0] mainly studied the virtual environments with deterministic transitions (e.g., Atari 11 or Mujoco [12), whereas sports games are real environments with stochastic game dynamics and complex context features. Moreover, player evaluation, as data-driven task, requires learning from a fixed dataset without exploration. The model must be able to handle the distribution shift experienced at test time.
To mitigate the impact distribution shift, Offline RL algorithms [3] typically strive for conservative policies that discourage visits Out-of- off--stributton (OoD) states by penalizing corresponding values. However, this approach cannot scaled to player evaluation since 1) our goal is not control (i.e., improving players' policy), but to RL analytical tool to evaluate observed actions in professional games, and 2) penalizing actions in OoD states distorts the evaluation.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

-100
100
42.5
Denety
Duratt
8
+
+
0.00
(d)
+
Denety
Denety
42.5
198.0
Figure 1: The predicted distribution of future goals in ice hockey game between Blues and Coyotes, 2018-19 NHL season. The shots are made in the positions (a) - (d), providing important motivations for risk-sensitive evaluation: Distributions (a) and (b) have the same expectation (around 0.6), but the first shot has a larger risk-averse estimate (at the confidence 0.8, we find 0.58 > 0.37) and smaller tisk-sekkig estimate the confidence 0.2, we find 0.68 < 0.77), and thus they have different impact on risk-sensitive evaluation. 2) Post-hoc calibration: the event of shooting from the position (d) (the back-court) rare an ice hockey game, and thus this event is likely be OoD, leading to a biased prediction at (d) (the predicted scoring chances are too large).
In this paper, we design an uncertainty-awart RL framework for risk-sensitive player evaluation. Figure[]showss a real-world example introducing our key motivations. Instead of directly influencing players' actions like other RI algorithms, we perform a post-hoe calibration of the learned action values. The main idea of our framework is to model important types of uncertainty in sports games:
1) Aleatoric uncertainty captures the intrinsic stochasticity of game dynamics caused by stochastic rewards, transition dynamies, and policies. We show that this stochasticity can be captured by distributional Bellman operator and propagated between action-value distributions by implementing Temporal- Difference (TD) learning with distributional RL.
2) Epistemic uncertainty is due to the finite training samples and OoD state-action pairs during testing. Online RL algorithms
overcome this uncertainty given sufficient exploration in the environment 8 However, when we have only a demonstration dataset with limited samples, the influence of epistemic uncertainty cannot be ignored. Striving for simplicity and scalability, we model the epistemic uncertainty with a Feature-Space Conditional Normalizing Flow (FS-CNF).
Based on the uncertainty estimations, we develop Risk-sensitive Game Impact Metric (RiGIM) for player evaluation. RiGIM filters the predictions for OoD samples and computes the impact of players' actions by conditioning on confidence level. Empirical evaluation shows that RiGIM is highly correlated with standard measures when compared to other baselines. We measure the accuracy of action-value predictions by matching them empirically with game results and evaluate the risk-sensitivity of RiGIM by its correlations with standard measures at different confidence levels.
Contributions. 1) We design uncertainty-awant RL framework that enables post-hoc calibrations on action values according to their aleatoric and epistemic uncertainties. 2) We demonstrate how the distributional Bellman operator captures the aleatoric uncertainty with action-value distributions from both a theoretical and an empirical perspective. 3) Striving for scalability, we design a feature-space density estimator that estimates the epistemic uncertainty with minimum overhead. 4) To the best of our knowledge, RiGIM is the first risk-sensitive metric that incorporates the inherent risk in environment dynamics into player evaluation. Although this work mainly focus professional sports games, our method is general and can be scaled to other stochastic environments or domains.
Related Works
In this section, we introduce previous works that are most related to our approach.
Uncertainty Estimation for RL. Uncertainty estimates have been widely used in RL for guiding exploration and stabilizing policies. To achieve this goal, an effective approach is to measure the
uncertainty of future returns: [4] designed an uncertainty Bellman equation that estimates the

---

15 16 17 predict the uncertainty of dynamies models and penalize the actions leading to uncertain returns. Instead of separately capturing the epistemic and aleatoric uncertainties, these methods often quantify the overall uncertainty with unified measure (e.g., variance entropy).
variance of the Q-value posterior distributions. Distributional RL methods 6 8 9 TO
directly model the distribution of future returns by computing corresponding quantities. Bootstrapped
DQN methods [88 19 20 21 learn ensembles of action-value Q functions to capture uncertainty.
Some following works 22 extend the Qennembll methods to offline RI. settings by learning from fixed dataset. Instead of focusing on the returns' uncertainty, an alternative approach is to measure the uncertainty of model dynamics: 124, [25] proposed model-based RL approaches that
Another line of approaches that heavily relies on uncertainty estimates is Risk-Sensitive RL (RSRL) [26]. The RSRL agents avoid states with high costs by estimating risk measure (e.g.. variance, Value at Risk (VaR), or Conditional VaR (27]). These algorithms (28 29 often learn a controlling policy based on a known MDP instead of an offline dataset (with unknown dynamics).
Player Evaluation. The most common approach to player evaluation is to quantify the impact of their actions on game results [30]. Previous works measured action impacts by predicting 1) whether goal will be scored within a fixed look-ahead horizon [3] 2) the change of winning chances [I], and 3) the expected number of goals within a possession [32]. Some recent works also trained action-valu Q-functions by dynamic programming [T] deep Sarsa (3 and Inverse RL 44. These methods compute an expected action value without modeling their potential risk, and they commonly assume the training and testing datasets are identically distributed.
3 Uncertainty-Aware RL framework for Player Evaluation
We represent the dynamics in sports games with a Markov Game model and introduce the motivation of estimating the aleatoric uncertainty and the epistemic uncertainty.
3.1 Finite-Horizon Markov Game Model
Player evaluation metrics commonly evaluate players by how much their actions influence the opportunity of scoring the next goal [ 34) Following this setting, we divide sports game into goal-scoring episodes, so that each episode begins immediately after goal (or at the beginning of the game), and 2) terminates when the next goal is scored (or the end of the game is reached). From an algorithmic perspective, this setting allows us to bound the support of future-goals distribution (Section 4.1 into [0, 1J, which leads to faster model convergence and more accurate evaluations.
For a scoring episode of length TH, we model its dynamies with a finite-horizon Markov game model [35]: G = (S, AA PT, R, O,TH, 7). At time step t E [0, TR). an agent k performs an action ag.t Ag a game state St E S after receiving an observation 0g E O. This process generates the next state St+1 ~ Pr(-|st, az) and a reward rk, = (st, ax is a discount factor. In this paper, we consider two agents k E (Home, Away) representing the home and away teams. The observed data D = (011, 4x,1, 11). (02, 02.2. T2), .... (ot, ak.t, ..)....] records the action ak.: performed by the team who possesses the puck. To alleviate the partial observability, game state includes the game history: 8, == (0t, at- o....... 00). The reward re is a 1-of-2 indicator vector that specifies which team (Home, Away) scores. We assign zeros to r, until a team scores at the end of an episode.
3.2 Uncertainty-Awer RL for Player Evaluation
Learning from Offline Data. As a data-driven behavior analytic tool, the player evaluation model assigns values to players' actions by learning from offline dataset. Under this setting, previous works [ [ 33 3 commonly assumed the training and testing datasets are sampled from the same underlying distribution. However, in practice, since the behaviors of players may change when some team members (especially core players or the coach) are traded or hired during a season, there is no guarantee that the visitation frequency of state-action pairs is consistent in different games. It is natural to assume a distributional shift between the games in the training and testing dataset: while the value function trained under one distribution, it will be evaluated on a different distribution.
Calibration with Uncertainty. To alleviate the influence of distribution shift, offline RI algo- rithms [3] commonly discourage the visit to OoD state-action pairs by lowering their values (36]
3

---

or penalising their rewards [24] or constraining the updated policy [37 22 players' performance, RL is employed as policy evaluation tool instead of controlling players. The trajectories in testing games correspond to observed players' actions that cannot be changed or influenced by penalties. Instead, we perform a post-hoc calibration of the predicted action values by modeling their epistemic uncertainty, which is due to lack of knowledge about Ool samples, and thus the resulting model is uncertain about the returns (Section 4.2). Since our goal is to develop risk-sensitive player evaluation metric, we estimate distributions of action values to model their aleatoric uncertainty, which is due to the intrinsic stochasticity in the game dynamics (Section 4.I). In practice, the quantification of aleatorie uncertainty be influenced by the epistemic uncertainty of input samples, so we filter the OoD samples by utilizing our density estimator (Section 5.1
However, to evaluate
Modelling the Uncertainty of Action Values
The aforementioned uncertainty-aware RL framework requires estimating the alcatoric and epistemic uncertainty for a risk-sensitive player evaluation. In this section, we introduce our distributional-RI approach for modelling aleatoric uncertainty and feature-space density estimator for measuring epistemic uncertainty (Figure shows the model architecture).
2
For apents
Quantiles
Away sccres
Home socres
Trace back the beginning play.
SPL-DON
Awnry
Away shotts
Adventaggees
Advaer Eagus
Action-Vblues
Jorw
Lotm
Isttn
FS-CNF
Density
Restet
ResNer
ResNet
(oa de)
Look ahead the nest goal.
Home Awayy
Game ends
Goal Event
3000
31000
3200
3300
3400
3500
3600
Game Time (In Seconds)
Figure 2: Model architecture. A play is turn where one team attacks and the other defends. We add Spectral Normalization ResNet outputs.
Figure 3: Illustrating the predicted distributions by showing the corresponding mean + standard deviation at each time step sports game.
4.1 Distributional RL for Capturing Aleatoric Uncertainty
Distributional RL learns the distribution of the random variable Z₁ (8t, 04) that corresponds to the number of future goals when a player of team i performs action a, in state 8;- In other words, can think of Zu (St, ax) as a random variable with outcomes corresponding to the sum of discounted rewards E 7 (g. A.), where S, 8., A, a,, S.-1 is distributed according to P- and A, is distributed according to T- (S.). Following the we represent the distribution of Z by uniform mixture of N supporting quantiles by 2x az) EN do,, (s),aa)) where Oz.: estimates the quantile at the quantile level (or quantile index) ft I++
(QR)-DQN method (6).
(1 < S N. and т, = i/N) and ōø,, denotes a Dirac distribution at 0x The model outputs [0x, 1, ......... N. are monotonically increasing quantile values computed with the spline DQN (SPL-DQN) by following [7].
Distributional Bellman Operator. When the player of a team k performs an action at state st the agent receives reward R3 (st, az) and moves to future state $141 ~ (y |s,, a,) where the agent's 's next action d++1 ~ π (AA41 SS41). This stochastic process can be captured by distributional
Bellman operator T (6]
T Zx a4) 2 Rs az) + (St1,,
where X A Y indicates that random variables X and Y follow the same distribution. Based on the distributional Bellman operator, we estimate the supporting quantiles of Z by minimizing the quantile Huber loss (with threshold n)):
N
p (r + B0 (88411 01+1) - 6 (St, az)) where
N

---

/200
|o| otherwise.
= -1-cco with Lo) = [n(ar| 1))
(2)
Illustration of Temporrl Projection. Figure 3 iillustrates the mean standard deviation of the action- values sampled from the predicted distributions (s, a), where and a follow the players' movements in match between the Flyers (Home team) and the Maple Leafs (Away team) on March 15, 2019. The figure plots values of the two output nodes. We highlight critical events and match contexts to show the context-sensitivig of our predictions.
We show that the predicted distribution of action values can measure the aleatoric uncertainty.
Proposition 1. Assume the Bellman consistency holds by 2 - R+ R+ Z where Z, R are vector-valued random variables and p" is the transition matrix of the stationary policy so P( (saa),s = P(s''la, s)T((')ss) s'), the uncertainty of action-value distributions Z under an entropy measure (-) can be given by:
H(2) = H - |A||S log(1 - -7)+ + +log (d))
(3)
1]|S|[A((SS|A is the induced matrix for distributions over state-action tuples by following policy and transition Pr.
where d² - [0,
The proof in Appendix B.I. Proposition disentangles the entropy of Z into 1) the entropy of reward variables that quantifies the uncertainty of current rewards, 2) the uncertainty induced by the discount factor, which determines how much the current uncertainty estimation should be influenced by the stochasticity of future rewards or transitions (i.e., small y reduces this influence), and 3) log-absolute determinant of the induced distribution matrix, which measures the amount of stretch or change that the transition function Pr and the policy apply to the initial state-action distribution.
Proposition demonstrates that the key components for representing the aleatoric uncertainty can be captured by Z when the Bellman consistency is reached by learning, which suggests the action-value distribution learned by distributional RL is an ideal estimator for aleatoric uncertainty. However, in practice, the estimation of Z cannot be well generalized to all samples because of insufficient exploration or limited training data, so we need to estimate their epistemic uncertainty.
4.2 Density Estimator for Capturing Epistemic Uncertainty
By definition, epistemic uncertainty stems from limited training data and is inherent to the model
fitting these data. common measure of epistemic uncertainty [0; y ar, D) [3
39
40 the
amount of information gained when the model 0 observes the true label of an input I. To estimate
this uncertainty measure, previous works [4] 42 4 utilized deep ensemble models and treated each ensemble as sample from the posterior p(0 D). However, adding additional deep ensemble layers to distributional RL model (i.e., as unified estimator for the joint distribution p(Z O D)) significantly increases the model complexity. Striving for model simplicity and scalability for large datasets, we build a feature space density estimator [40] to detect OoD samples.
In this work, we design Feature Space Conditional Normalizing Flow (FS-CNF) to estimate sample density in the training distribution with a minimum overhead. The main components are:
Feature Extractor. FS-CNF shares the same feature extracting layers with the distributional RL models. Note that common reason why traditional feature extractors might fail to capture epistemic uncertainty is feature collapse 44, which maps OoD samples to iD regions in feature space. To prevent this phenomenon, ideal feature extractor So must be subjected to bi-Lipschitz constraint:
| - .22l 1| 2 [[f((z)) - fo(x2)]lr 2 32||a1 - x2 12 Ven, x2 E D
(4)
where 1) the lower Lipschitz bound ensures sensitivity to distances in the input space (i.e., sensitive to OoD samples) and 2) the upper Lipschitz bound ensures smoothness in the feature space (i.e., prevents overfitting to the input variations). To ensure this bi-Lipschitz condition in practice, follow 44 and use residual networks with spectral normalisation as our feature extractor.
Density Estimator. Based on the extracted features, FS-CNF utilizes the Masked Auto-regressive Flow (MAF) 45 design that estimates the density of input variables the training data distribution with ruto-regressive constraint. Additionally, to make FS-CNF more sensitive to the abnormal
5

---

predictions in OoD samples, the density model is conditioned on the expected returns of action- value distributions, so p(z Æ = E; 1( =F) where =Æ [ a)]]==]]]])) k= and a (o, a) (we use o instead of since large input dimension influences the accuracy of estimation). We implement p(z, 1 E )=== = (2r,lus (exp(a:))²) where μ4, = 16y (21::-11 2x) and 0g
Vo (11 Zg). The neural function v5 is implemented by stacking multiple MADE layers [46].
5 Player Evaluation
In this section, we introduce our player evaluation metric and risk-sensitive rankings.
5.1
Risk-sensitive Impact Metric
Mcasuring Risk. We measure the risk of player's action with the aleatoric uncertainty since modeling the intrinsic stochasticity of the game dynamics consistent with the goal of sports analytics. However, as Figure 4 shows, when we quantify the alcatoric uncertainty with predictions from distributional RL, the perfor- mance influenced by the density of input sam- ples (i.e., OoD samples have lower accuracy).
6000
0.04
4000
2000
0.01
0.00
Figure 4: We discretize the ice hockey rink into 5x5 regions. For each region, the left heatmap shows the number of shots in the train- ing dataset, and the right heatmap shows the Mean Absolute Error (MAE) between the esti- mated and the real aleatoric uncertainty for shot values in the testing dataset. We observe nega- tive correlation (-0.761) between the density and the MAE across these regions.
An effective approach to verify whether 24 (s, a) accurately captures the true aleatoric uncertainty to check the epistemic uncertainty for each input data (s, a) [40]: 1) a high input density (p(-Izg) 2 e) indicates low epistemic uncer- tainty, (i.e., (s, a) inD) and we can trust the aleatoric uncertainty estimated by distributional RL. 2) a low input density (p(-Izg) < e) indi- cates high epistemic uncertainty (i.e., the input OoD, and do not use the model prediction). In practice, e determined based on the validation dataset.
Risk-Sensitive Action Impact. The impact Φ(s,a) measures how much an action a changes the future return of player's team. In terms of the value function, this is the change action value due to player's s action. Previous works N 2 computed the impact with the expected next-goal return ((s,, (t) BiZ(s, a,)]. Q(-) does not take into account the inherent variability of the returns and thus cannot estimate the risk of an action. To understand how players respond to risk, we propose Risk-sensitive Game Impact Metrie (RiGIM) based on Z (s, a) and p(-1z)):
1e(86+1,04-113 (81+1, (4+1) (0 ):) [((x))2 RiGIM((c)
Σ al) as.
(5)
(s.0)ED)
where e E [0, 1] is the confidence level, 2o denotes the (1 - c) quantile in Z(-) and n(s, a, denotes the number of times that player performs action a at a state in the testing dataset D'. We omit the term r; since r, = except at scoring step T, and 8T is the terminal state scoring episode. eb-) can either be 1) risk-averse (with large c), with better sensitivity to bad outcomes or 2) risk-seeking (with a small c), with better sensitivity to positive outcomes.
5.2 Case Study: Player Ranking in Testing Games
We rank players according to their RiGIM scores in the NHL testing games (see the experiment setting in Section (6) at different confidence levels. Tables Tindaiilustrates how a domain expert could use our method to gain insight into which types of players exhibit different risk-taking behavior. Table shows a risk-seeking ranking (confidence e = 0.2), which favors offensive players (e.g.. Centres (C)) with strong scoring ability. Aleksander Barkov, who scores the most points in these games, captured by this ranking. When we set confidence to 0.8. Table shows risk-averse ranking which highlights players in defensive positions (e.g., Defensive (D)). John Klingberg, the

---

defenseman with the most assists, is listed in the top-10 players. We believe the differences between Tables and 2 can be explained by the fact that RiGIM is correlated with the types of actions player performs. Intuitively, in ice-hockey, some actions are more risk-seeking (shot) in terms of scoring while other actions are risk-averse (carry or pass). general, players in the backcourt (e.g.. defensemen) are more likely to perform risk-averse actions that have smaller variance on the scoring chances.
Table I: Top 10 players with confidence 0.2.
Table 2: Top 10 players with confidence 0.8.
Player
Position
Team
RiGIM
Player Name
Position
Team
RiGIM
Jonathin Toews
CHi
14.72
Radek Faksa
C
DAL
2.74
Anze Kopitar
LAK
14.55
Leon Draisaitl
C
EDM
8
2.51
Vincent Trocheck
C
FLA
14,02
John Klingberg
D
DAL
2.46
Tomas Hertl
C
SIS
13.97
Esa Lindell
D
DAL
2.29
John Tavares
C
TOR
13.92
Connor McDavid
C
EDM
2.23
Tyler Seguin
C
DAL
13.71
Tomas Heril
C
SIS
1.93
Leon Draisuiil
C
EDM
13.16
Miro Heiskanen
D
DAL
1.86
Aleksander Barkov C
FL.A
12.63
Elias Pettersson
C
VAN
1.79
Srun Couturier
PHI
12,62
Tyler Seguin
C
DAL
1.78
Nathan MacKinnon
COL
12.48
Roope Hintz
LW
DAL
11
4
1.77
Empirical Evaluation
Dataset. Our experiments utilize both ice-hockey and soccer dataset from the National Hockey League (NHL) and major European soccer leagues, which contain 9,213,371 events, covering 195 teams, 4,172 games, and 6,513 players. These datasets consist of events around the ball. Each event records the identity and action of the player possessing the ball, with time stamps and features of the game context (see all the game features in Appendix A. To the best of our knowledge, this is the most extensive study for player evaluation. Note that we do not utilize virtual environments like Atari [11 or Mujoco [12] because 1) the dynamics in these environments are deterministic without uncertainty to be modeled and 2) this paper mainly studies sports games that have stochastic dynamics 30 which are valid test-beds for our method.
Experiment Settings. We divide the dataset into a training set (80%), a validation set (10%), and a testing set (10%) according to game dates, so that games in the testing set happened after the games in the training and validation set. To predict the action values in the testing games, the metric must remain robust to OoD data points. We report the results averaged over 5 independent runs.
Comparison Methods. We employ an ablation design that iteratively removes parts from RIGIM. GIM removes the uncertainty estimator by directly using Deep Recurrent Q-Network for estimating action values 22]. TO-GIM removes the recurrent model and uses a Deep Q-Network (DQN) for the value function. We then replace the RL framework with supervised learning framework for estimating action values by following VAEP 3 Instead of using function approximators, Scoring Impact (SI) [1] implements the tabular- based value iteration algorithm for computing action values from discretized spatial and temporal features. Expected-Godd (EG) metric directly uses the expected action values instead of impact values for measuring player performance. The last metric Plus-Minus (+/-) is based on game statistics and measures the goal-gain with and without the player on court. We summarize these metrics in Table 3
Table 3: A summary of the baseline Methods for player evaluation.
To study how well SP-CNF boosts model perfor- mance, we compare 1) a Gaussian Discriminant Analysis (GDA)-RiGIM metric that replaces SP-CNF with GDA [40], and 2) Naive (Na)- RiGIM metric that removes the epistemic un- certainty estimator and uses all the predicted distributions to compute players' impact (see Appendix A.2 and A.3 for more details.).
Method
Risk-
History-
RL-
Contimoou
Inguct- Bised
Contest-
x
EG
x
x
x
x
x
VAEP
x
6.1 Player Evaluation Performance: Correlations with Standard Measures
We follow 2 and compute the correlations between player ranking metrics and standard measures on the testing games in a game season, because 1) the player (or agent) evaluation task has no ground-truth labels or rewards to maximize, 2) the correlation to all measures (including penalty measures) can measure whether the metrics can form comprehensive evaluation to player's overall
7

---

performance. We study 11 success measures for ice hockey and 8 success measures for soccer. To make the results more comprehensive, we also add 5 penalry measures. The studied measures are popular measures from the NHI. and soccer statistics websited] Following the popular risk-measures like Conditional Value at Risk (CVaR), we treat e as hyper-parameter. Since the comparison metrics, we study the options of 1) setting the confidence level (c) of RiGIM to 0.5 for a fair comparison 2) empirically determining с with the validation set (c* = 0.34 and 0.49 for the ice hockey and soccer datasets). The risk-sensitive results are shown in Section 6.2
methods are expectation-basee
Tables 4 and [ show the average correlations in 5 independent runs on the testing dataset (see Tables C.I and C.2 in Appendix for the complete mean + standard deviation results). RiGIM achieves the highest correlations with 14 out of 19 success measures and the smallest correlations with 3 out of 5 penalty measures in ice hockey and soccer games. This observation shows that RiGIM is a comprehensive metric that can detect both the positive and the negative influence of player. If we remove SP-CNF or replace it with other uncertainty estimators, most correlations become weaker except for the correlations with the SHP and SHG measures. This is because SHP and SHG rarely happen in season (scoring with fewer players ice is difficult). SP-CNF detects this phenomenon and assigns small densities to these rare events. RiGIM filters the event with a small density (see Equation 5T. which might cause the loss of information and make its correlation with SHP and SHG less significant. SI correlates well with goal measures (Goals and Game Winning Goals) but has relatively poor correlations with other measures. This is because assigning adequate value for all actions, including those with only intermediate effects on goal scoring, requires credit propagation over longer sequences, where neural nets are better at credit propagation than discretizing and using tabular representation. For other risk-neutral methods, their performance is generally less satisfying when compared with risk-aware methods, especially for the +/- metric, which shows the importance of capturing risk and modeling the context features.
Table 4: Correlations with standard measures in the ice hockey games. The success measures are assist, goal, Game Winning Goal (GWG), Overtime Goal (OTG), Short-handed Goal (SHG),
Power-play Goal (PPG), Point (P), Short handed Point (SHP), Power-play Point (PPP), Time On Ice
(TOI), and Shots (S). The penalty measure is Penalty Minute (PIM).
Methods
Assist
Goal
GWG
OTG
SHG
PPG
Point
SHP
PPP
TO1
8
PIM
0.181
0.189
0.187
0.028
0.071
0.063
0.206
0.119
-0.071
0.021
0.038
-0.014
BG
0.239
0.303
0.130
.0.053
0.023
0.226
0.153
0.534
40.112
SI
0.237
0.596
0.409
0.123
0.095
0.351
0.452
0,066
0.274
0.224
0.405
0.138
VAEP
0,238
0.454
0.225
0.06
0.053
0,326
0,382
-0.0
0,321
0.086
0.362
0.027
TO-GIM
0.397
0.394
0.139
0.16
0.151
0.216
0.455
0.153
0,295
0.356
0.387
0.058
GIM
0.456
0.408
0.167
0,158
0.134
0.246
0.501
0.137
0,345
0.395
0.431
0.061
Nā- RIGIMO
0.593
0.376"
"0.225
0.173
0.IS2
"0,315
0.825
'0.775
0,453
"0.997
0.60
'0111
GDA-RIGIM(O.5
0.591
0.475
0.221
0.174
0.152
0.315
0.623
0.174
0.452
0,593
0.609
0,113
RIGIM(OL5)
0.675
0.477
0.266
0.184
0.11
0.355
0.678
0.141
0.529
0..88
0.7
0,146
RiGIM(c*)
0.68
0.477
0.269
0.187
0.107
0.357
0.681
0.141
0,531
0.685
0.707
0.147
Table 5: Correlations with standard measures in the soccer dataset. The success measures are goal, assist, Shots per Game (SpG), Pass Success percentage (PS%), Key Passes (KeyP), Dribbles (Drb), Crosses and (being) Fouled. The penalty measures are Yellow (Yel) and Red Card Received, Offsides
OOff) and Own Goals (OwnG).
Methods
Goal
Assist
SpG
PS%
KeyP
Drb
Crosses
Fouled
Yel
Red
Off
OwnG
0.284
0,318
0.199
0,288
0.218
0.119
0.017
0.035
0.001
-0.099
0.053
-0.001
EG
0.422
0.173
0.328
0.164
0.278
0.013
0.040
-0.026
0.534
0.034
-0.124
-0.008
SI VAEP
0.585
0.153
0.438
-0.140
0.052
0.050
0.216
-0.065
0.114
-0.089
-0.249
-0.102
0.093
0.290
0.121
-0.111
0.116
0.059
0.082
-0.00
0.024
0.133
-0.055
-0.051
TI-GIM
0.614
0.455
0.715
0.148
0.431
0.161
0.355
-0.007
-0.027
-0.346
-0.168
GIM
0.627
0.462
0.72
0.149
0.473
0.437
0.169
0.358
-.00
-0.025
-0.336
-0.154
'0.46
0.507
'0.741
00.14
'0333"
"0.A5
'O177""
"0,30
"0100"
0.007
--0.309
-0.
0.649
0.506
0.725
0.132
0.478
0.161
0.389
0.147
0.018
-0.259
-0.125
RiGIM(0,5)
0.671
0.577
0.756
0,181
0.574
0.530
0.239
0.448
-0.092
-0.039
-0.451
-0.185
RiGIM(c)
0.682
0.583
0.757
0.186
0.575
0.531
0.238
0.446
-0.101
-0.022
-0455
-0.14
http://www.nhl.ce http://www..ll..oo///ttttt////a/ and httt:p/s//www..oeeeerddd.oo////s
8

---

6.2 Sensitivity to Risk: Correlations Conditioning on Different Confidence Levels
We measure whether RiGIM is sensitive to the risk by its correlations with the standard measures, where RiGIM is conditioned on specific confidence level e (from 0 to 1), for example, (cGIM(c). which indicates with probability that the players' impact should be at least RiGIM(c).
Figures and show the correlations at different confidence levels for ice-hockey and soccer games. RiGIM is sensitive to risk, in the sense that it has different correlations with standard measures at these confidence levels, whereas GIM, as iis--nettral metric, is unaware of the risk, and thus its correlations remain unchanged. Compared to other baselines, our RiGIM generally maintains higher correlations with success measures and lower correlations with penalty measures. The exceptions are the correlations with the SHP, SHG, and OTG. For the same reason as discussed above, SP-CNF may filter them during testing. We find when becomes smaller, RiGIM(c) becomes risk-seeking, and thus achieves a higher correlation with success measures. However, the correlations drop when c approaches O. This is because Z% denotes the estimates at the largest quantile level (see Equation 5 which corresponds to the most optimistic estimation action value (i.e., label scoring for all the shots). The overly risk siek-neeiing estimation can induce a mismatch between estimated values and game facts, and thus cannot reflect the real contributions of players.
Dixee unj Dad POMEO
Time (011))
Stxt Hocced Dad 19-00
Plese Play Oe IPASE
Oonkatos
0.58
012
335
0.50
0.75
10
9:0
625
0.50
035
1N
0.30
335
0.50
0.75
26
0.3
025
0.50
0.5
3.0
0.00
0.25
E50
0.75
0.30
235
0.50
0.75
3.00
danon
Conridenno
Conkddrcca
Confishnnoe
Caniidence
Polt:
Texade 2011
Pin (PM)
0.5
Orinattnn
0.33
233
0.50
0.75
300
025
0.50
035
1
0.30
335
0.50
0.75
36
3.3
023
0.50
0.5
3.0
0.00
3.25
2.00
0.75
0.00
235
0.50
0.5
3.00
Coviddenee
Qonidernc
Conidance
Figure 5: Correlations (Mean + standard deviation) with success measures (the first 11 plots) and penalty measures (the last plot) at different confidence levels in ice-hockey games.
Goal
Assiit
Shets garne Bpcd
Fasz Success Parcertope (P3)
Pussea Per Garee HRof!
Ortckis per gara
0.79
0.5
33
Oia
loca
0.3
0.
Cie local
335
C75
Oraso MAT BUTN
Of 100 Mer
Own pools
222
1.5
222
03:
3.04
0.50
625
4.50
335
0.50
6.25
0.50
3.M
0.00
0.35
0.75
Figure 6: Correlations (Mean + standard deviation) with success measures (the first 8 plots) and penalty measures (the last plots) at different confidence levels for in soccer games.
6.3 The Prediction Accuracy: Match Future Scoring Frequencies With Action- Action-Values
We study how well the predicted action-value distributions match the real next-goal scoring frequen- cies under discrete game contexts. These game contexts are constructed by dividing the continuous state space into discrete bins. To calculate the empirical scoring frequency associated with each bin, we assign an observed state to a bin B according the values of three discrete context features in the current observation: Manpower Differential, Goal Differential, and Period. The empirical and estimated scoring probabilities for a bin (with size [B) are defined follows:
a) Empirical Scoring Chances: for each 8 E B, we set ga (s) if the observed scoring-episode containing state ends with a goal by team k. Then G (B) = [B] E.e 0h (s).

---

b) Estimated Scoring Chances: for each s E B, given N samples from the calibrated distribution (x(s) Zz(s, =)))(===))2=++ the estimated chances are: Gx N Eeen En (s).
Table 6:
The difference between the empirical and the estimated scoring chances in different contexts.
Results are averaged over 5 runs = standard error. 1 (4) indicates that difference is statistically greater (smaller) than the difference achieved by RiGIM with p-value I 0.01 according to the Wilcoxon signed rank test.
Ice-Hockey
Soccer
Manpower
Shert- Handed
Even-
Power-
Short- Handed
Power-
Differential
Strength
Play
Strength
Play
0.078
0.09 0.082
1.099 0.085
0.014
0.05 1
0.037
Na-RiGIM
0.133 + 0.016
0.064 + 0.016
0.013 + 0.009
0.226 + 0.019
0.136 0.019
0.175 + 0.028
GDA-RiGIM
0.148 + 0.035
0.072 + 0.029
0.017 0.011
0.216 + 0.022
0.013
RIGIM
0,080
0.020
0,058
0.008
0.047 4: 0.046
0.204
0.005
0.133
0.007
0.147 +: 0.033
Dillerential
GIM
0.238 1 0.122
0.108 1 0.084
0.271 1 0.039
0.155 F 0.047 4
0.155 + 0.054 1
0.221 + 0.149
Na-RiGIM
0.238 + 0.006
0.045 + 0.015
0.108 + 0.031
0.157 + 0.02
0.104 0.024
0.16 +
GDA-RIGIM
0,236 + 0.007
0.045 + 0.016
0.11 0.027
0.165 + 0.018
0,117 = 0.017
0.175 = 0.007
RIGIM
0.193 = 0.021 3
0.029 +: 0.015 2
0.092 + 0.019
0.152 +: 0.008
0.109 = 0.004
0.149 +: 0.013
Period
2 half
1 halt
N/A
GIM
0.095 0.055
0.111 1 0.086
0.114 1 0.084
0.191 0.037
0.104
0.059 1
Na-RiGIM
0.139 0.018
0,044 + 0.015
0,024 + 0.015
0.237 + 0.013
0.061
+ 0.03
GDA-RIGIM
0.143 = 0.028
0.050 + 0.025
0.033 + 0.025
0,238 + 0.012
0.059 = 0.026
RIGIM
0.032 0.005
0,014 0.009
0.226 0.011
0.058 0.009
Table shows the average absolute difference between & and G* based on 5 independent runs. We implement a Wilcoxon signed rank test 44 to study whether the predictions from baseline methods are different from that of RiGIM for all samples. Our baselines are the learning-based action-values metries for the context-aware evaluation. RiGIM achieves a minimum distance in 12 out of bins. This is because 1) FS-CNF outperforms GDA by computing more accurate uncertainty estimator for filtering OoD states and 2) the distribution estimates contain richer information than expectation estimates, allowing (-) to better match the scoring frequencies than Q(-). This observation is consistent with the findings in (6].
6.4 Limitations
A Study for Sports Games. This work uses stochastic sports games as the testbeds for uncertainty estimation, but we argue the same methods can be easily migrated to other applications in stochastic environments such as autonomous driving or healthcare. Our approach of measuring aleatoric and epistemic risks can be adapted offline RL for learning conservative policies.
Evaluation Instead of Control. Our method focuses on player evaluation instead control. We believe both tasks challenging with respect to different aspects. Evaluation requires the action values to accurately reflects an agent's real contribution to game-winning, An ideal evaluation metric can provide in-game predictions of game outcomes, which is important for the sports industry.
Conclusion and Future Work
In this paper, we designed an RL framework for quantifying the alcatoric uncertainty and the epistemic uncertainty from stochastic sports datasets. This framework enabled distributional RL and an FS-CNF model to estimate both uncertainties, with which we proposed risk-sensitive evaluation metric RiGIM Empirical results show that RiGIM correlates well with success measures and the correlation sensitive different risks. direction of future work is extend our model to other domains.
Acknowledgments and Disclosure of Funding
We acknowledge the funding from the Canada CIFAR AI Chairs program, and the support of the Natural Sciences and Engineering Research Council of Canada (NSERC). Resources used in this work were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute https::/vectorinstiioooo http:/orrinnttuutt...aaaaaeeess ai/partners/
10

---

References
[1] Kurt Routley and Oliver Schulte. A markov game model for valuing player actions in ice hockey. In The Conference Uncertainty in Artificial Intelligence (UAI), pages 782-791, 2015.
[2] Guiliang Liu and Oliver Schulte. Deep reinforcement learning in ice hockey for context-aware player evaluation. In International Joint Conference Artificial Intelligence (IJCAI), pages 3442-3448, 2018.
[3] Tom Decroos, Lotte Bransen, Jan Van Haaren, and Jesse Davis. Actions speak louder than goals: Valuing player actions in soccer. In The 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1851-18611 2019.
[4] Yudong Luo, Oliver Schulte, and Pascal Poupart. Inverse reinforcement learning for team sports: Valuing actions and players. In International Joint Conference on Artificial Intelligence (IJCAI), pages 3356-334 3363, 2020.
[5] Mare G. Bellemare, Will Dabney, and Mark Rowland. Distributional Reinforcement Learning. MIT Press, 2022.
[6] Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforce- ment learning. In International Conference Machine Learning (ICML), volume 70, pages 449 458, 2017.
[7] Will Dabney, Mark Rowland, Mare G. Bellemare, and Rémi Munos. Distributional reinforce- ment learning with quantile regression. In AAAI Conference on Artificial Intelligence, pages 2892-2901, 2018
[8] Borislav Mavrin, Hengshuai Yao, Linglong Kong, Kaiwen Wu, and Yaoliang Yu. Distributional reinforcement learning for efficient exploration. In International Conference on Machine Learning (ICML), volume 97, pages 4424-4434, 2019.
[9] Fan Zhou, Jianing Wang, and Xingdong Feng. Non-crossing quantile regression for distribu- tional reinforcement learning. In Neural Information Processing Systems (Neurips), 2020.
[10] Fan Zhou, Zhoufan Zhu, Qi Kuang, and Liwen Zhang. Non-decreasing quantile function network with efficient exploration for distributional reinforcement learning. In International Joint Conference on Artificial Intelligence (IJCAI), pages 3455-3461, 2021.
[11] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-276 2013.
[12] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, IROS, pages 5026-5033, IEEE, 2012.
[13] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. CORR, abx/2005.01644 2020.
[14] Brendan O'Donoghue, Ian Osband, Rémi Munos, and Volodymyr Mnih. The uncertainty bellman equation and exploration. In International Conference on Machine Learning (ICML), volume 80, pages 3836-3845, 2018.
[15] Yunhao Tang and Shipra Agrawal. Exploration by distributional reinforcement learning- In International Joint Conference on Artificial Intelligence (IJCAI), pages 2710-2716, 2018.
[16] Shangtong Zhang and Hengshuai Yao. QUOTA: the quantile option architecture for reinforce- ment learning. AAAI Conference on Artificial Intelligence, pages 5797-5804, 2019.
[17] Yudong Luo, Guiliang Liu, Haonan Duan, Oliver Schulte, and Pascal Poupart. Distributional reinforcement learning with monotonic splines. In International Conference on Learning Representations (ICLR), 2022.

---

[18] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. In Neural Information Processing Systems (Neurips), pages 4026 4034, 2016.
[19] Richard Y. Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. UCB and infogain exploration via q-ensembles. CoRR, abs/1706.01 2017.
[20] Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep rein- forcement learning. In Neural Information Processing Systems (Neurips), pages 8626-8638, 2018.
[21] Felipe Leno da Silva, Pablo Hernandez-Leal, Bilal Kartal, and Matthew E. Taylor. Uncertainty- aware action advising for deep reinforcement learning agents. In AAAI Conference on Artificial Intelligence, pages 5792-5799, 2020.
[22] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy q-learning via bootstrapping error reduction. In Neural Information Processing Systems (Neurips), pages 11761-1177,, 2019.
[23] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- based offline reinforcement learning with diversified q-ensemble. In Neural Information Processing Systems (Neurips), pages 7436-7447, 2021.
[24] Tianhe Yu, Garrett Thomas. Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: model-based offline policy optimization. In Neural Information Processing Systems (Neurips), 2020.
[25] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. In Neural Information Processing Systems (Neurips), 2020.
[26] Oliver Mihatsch and Ralph Neuneier. Risk-sensitive reinforcement learning. Machine learning, 49(2):267-290, 2002.
[27] Tyrrell Rockafellar, Stanislav Uryasev, et al. Optimization of conditional value-at-risk. Journal of risk,
2000.
[28] Yun Shen, Michael J. Tobia, Tobias Sommer, and Klaus Obermayer. Risk-sensitive reinforce- ment learning. Neural Computing, 26(7):1298-1328; 2014.
[29] Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision- making: a cvar optimization approach. In Neural Information Processing Systems (Neurips), pages 1522-1530. 2015.
[30] Jim Albert, Mark E Glickman, Tim B Swartz, and Ruud H Koning. Handbook of Statistical Methods and Analyses in Sports. CRC Press, 2017.
[31] Peter Xenopoulos, Harish Doraiswamy, and Cláudio T. Silva. Valuing player actions in counter- strike: Global offensive. In IEEE International Conference Big Data (IEEF BigData), pages 1283-1292. IEEE, 2020.
[32] Daniel Cervone, Alex D' Amour, Luke Bornn, and Kirk Goldsberry. A multiresolution stochastic process model for predicting basketball possession outcomes. Journal of the American Statistical Association, 11 (1(14))585-599 2016.
[33] Guiliang Liu, Yudong Luo, Oliver Schulte, and Tarak Kharrat. Deep soccer analytics: learning an action-value function for evaluating soccer players. Data Mining Knowledge Discovery, 15(5):153 2020.
[34] Xiangyu Sun, Jack Davis, Oliver Schulte, and Guiliang Liu. Cracking the black box: Distilling deep sports analytics. In The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3154-3162, 2020.
[35] Michael L.. Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine Learning, Proceedings of the 11th International Conference, pages 157-163, 1994.
12

---

[36] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Neural Information Processing Systems (Neurips), 2020.
[37] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning ICML, volume 97, pages 2052-2060 2062, 2019.
[38] Lewis Smith and Yarin Gal. Understanding measures of uncertainty for adversarial example detection. In The Conference on Uncertainty in Artificial Intelligence (UAI), pages 560-569, 2018.
[39] Joost van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using single deep deterministic neural network. In International Conference on Machine Learning (ICML), volume 1 119, pages 9690-97000 2020.
[40] Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip H. S. Torr, and Yarin Gal. Deterministic neural networks with appropriate inductive biases capture epistemic and aleatoric uncertainty. CoRR, abs/2102. 1022211...1. 2021.
[41] Balaji Lakshminaray Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Neural Information Processing Systems (Neurips), pages 6402-6413, 2017.
[42] Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: alternative approach to efficient ensemble and lifelong learning. In International Conference on Learning Representations (ICLR), 2020.
[43] Michael Dusenberry, Ghassen Jerfel, Yeming Wen, Yi-An Ma. Jasper Snoek, Katherine A. Heller, Balaji Lakshminarayanan, and Dustin Tran. Efficient and scalable bayesian neural nets with rank-1 factors. International Conference on Machine Learning ICML., volume 119, pages 2782-2792, 2020.
[44] Joost Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using a single deep deterministic neural network. In International Conference on Machine Learning ICML, volume 119, pages 9690-9700 PMLR, 2020.
[45] George Papamakarios, lain Murray, and Theo Pavlakou. Masked autoregressive flow for density estimation. In Neural Information Processing Systems (Neurips), pages 2338-2347, 2017.
[46] Madson L. D. Dias, César Lincoln C. Mattos, Ticiana L. Coelho da Silva, José Antônio Fer- nandes de Macêdo, and Wellington C. P. Silva. Anomaly detection in trajectory data with normalizing flows. In International Joint Conference on Neural Networks (IJCNN), pages 1-8, 2020.
[47] William Jay Conover. Practical nonparametric statistics, volume 350, john wiley sons, 1999.
13

---

8 Submission of papers to NeurIPS 2022
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Check Section 6.4
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Check our Appendix D.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Proposition I and the proof in Appendix B. (b) Did you include complete proofs of all theoretical results? [Yes] Check Proposition and the proof in Appendix B.
Check
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URLY? [Yes] Check detailed dataset introduction in Appendix A.1 and the instructions Appendix A.2.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] Check the experiment settings Section 6 and Appendix A.2.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We use 5 random seeds for all experiments. Check the experiment settings Scction 6 our experiment results and Table C.I and C.2 in Appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Check Appendix A.4.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] There is not formal citation, so we add the link to the data source. Check our dataset introduction in Appendix A.1.
(b) Did you mention the license of the assets? [N/]] There is no such licence that we can cite for this dataset.
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] We have corporated on many projects.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes] Check our dataset introduction in Appendix A.I.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "YugNU84IUAw", "reviewer": "Reviewer_tcWQ", "summary": "This is a new review, since my previous review was written for the wrong paper. My apologies for the mixup.\n\nThe paper introduces a methods based on RL for player evaluation in ice hockey and soccer. The method is called RIGIM\nDistributional methods previously used in Atari and Mujoco are now used to evaluate real games.\nThe main contribution is the introduction of the RIGIM method. The description is quite technical and does not provide intuition how it works, what the short comings are, and how it can be improved.\nAn evaluation is provided, and a comparison to other methods.  The method has a low correlation to standard measures, but appears to perform somewhat better than other methods. \nThe paper does not really provide a discussion on the reasons for this.\nMy main problems with the paper are the clarity of the message and the clarity of the analysis.\nIt contains a description of a lot of work done, but what is the problem that is addressed? What is the contribution to the field? Why is it important?\n\nThanks very much for the message and for the request for clarification.\n\nThank you for the question. I can emphatize with the authors, and will elaborate on how the paper can be improved and why.\n\nExamples: Section 4 is a technical section, that is missing an intuitive description of their method, and that is missing a high level explanation. Can the authors please add to rheir techjical description a high level explanation of the problems of related work that is adressed by their method. What is different, and why does it work better?\n\nExamples of where the explanations can be improved; In Section 6.1 the authors write: “If we remove SP-CNF or replace it with other uncertainty estimators, most correlations become weaker except for the correlations with the SHP and SHG measures.” Can you please explain or speculate why this is the case?\n\nSection 6, and Table 4 and 5 show correlations that are rather low, yet section 6.2 and section 6.3 fail to address this issue. The authors write: “RiGIM(c) becomes risk-seeking, and thus achieves a higher correlation with success measures. However, the correlations drop when c approaches 0. This observation is consistent with the fact that an overly risk-seeking estimate cannot reflect the real contributions of players.” Why is that so, why can it not reflect the real contributions?\n\nThe authors may also consider adding an explicit problem statement and revising their contributions to address questions of the field. As they are, the contributions are a list of work that has been done, failing to address the importance and which insights are contributed to the field.\n\nOverall, I challenge the authors to let their descriptions transcend the description of what they have done, and provide explanations as to the why.", "questions": "Please provide a better explanation how the method works, how it builds on other work, how it compares to other work, what the strengths and weaknesses compared to other work are, and please provide better ideas on how to take this further. \nThe overall correlations to stnadard measures are quite low, why is that? What can be done to improve?", "limitations": "No, the limitations are listed very briefly, without real substantiation.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \nnew method for player evaluation\nThe method can learn from offline data\nUse of distributional RL is interesting\n\nWeaknesses:\nPerformance correlates poorly with standard measures, although somehat better than other methods\nNo real insight provided how the method works, and how it can be improved", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DQLULXMQTh", "reviewer": "Reviewer_kCNJ", "summary": "The authors develop an offline RL model for risk-sensitive agent evaluation and apply it to study player impact in sports games. Unlike previous approaches, their method RiGIM is sensitive to stochasticity in the environment (some situations have lower/higher variation in their outcomes) and can handle the OoD actions that plague Offline RL. Because they are focused on evaluation (and not control), the typical techniques for dealing with OoD actions (lowering action values or constraining the policy) will not work and so the authors develop a scalable density estimator based on normalizing flows. These techniques are evaluated on a large dataset of hockey and soccer games and correlated with traditional measures of impact. ", "questions": "Can you give some intuitions for the case study results in Table 1/2. The only comment is that there are more defensemen in the higher confidence top 10 -- although this is true empirically -- why is this a prediction of higher confidence? It doesn't seem to clearly correlate with the other metrics shown. Tyler Seguin and Connor McDavid have similar stats and play in the same position so what accounts for them being in the different top 10 lists? The authors mention a couple of qualitative findings e.g., \"low confidence favors centers, strong scoring ability) but its not clear what is driving these predictions. How would a layperson use this tool?\n\nFor tables 4/5/6, why was c set to 0.5 instead of fit on the validation set? How would one decide on a value for c in practice?\n\nIn table 6, many of the bolded numbers are not different at a level of statistical significance. Ideally, the algorithm can be run more times for the camera ready to reduce the uncertainty about performance. \n\nWhy is the correlation with traditional metrics a good way of measuring success? The \"success measures\" are very simplistic features of the game so it could be that a better correlation with these measures is actually a signal that the algorithm is doing something naive rather than sophisticated. \n\nWhy do traditional metrics better correlate with risk-seeking versions of (smaller values of c)? \n\nWill the post-hoc calibration techniques developed in this work also apply to the offline RL control setting? Might this be a better approach than constraining the actions that the agent can take? ", "limitations": "Limitations are addressed. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is clearly written with consistent notation and the problem is well articulated. Both the problem statement and the techniques used to solve it are novel to my knowledge. Furthermore the approach is demonstrated on a large dataset of practical importance with potentially promising results. The main weaknesses are related to the clarity and interpretation of the evaluation (expanded on in the Question section). The evaluation of the results are confusing and it is not clear how this technique should be measured or used. \n\nSmall issues: \n\nLine 37-39 citation of what algorithms are being contrasted here.\nWhat is the x-axis supposed to represent in (a)-(d) in Figure 1? \n\nGrammar: \"We measure whether RiGIM is sensitive to the risk by its correlation\" (Line 287). \n\nLines 294-296 are repetitive with 282-283", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5rax539skI", "reviewer": "Reviewer_mdqR", "summary": "The paper addresses the problem of evaluating player behavior in games based on the expected value and uncertainty of actions taken, specifically when examining data obtained from live sports (ice hockey and soccer). \n\nThe authors propose a model to decompose games into a series of episodes with inherent stochasticity (\"aleatory uncertainty\") and limited observational data (\"epistemic uncertainty\"). The model uses distributional reinforcement learning to address stochastic dynamics. The model also filters samples using a density estimator to address out-of-distribution data and offline data sampling sparsity. Evaluation results show better correlations to game metrics (like scoring goals) when compared to alternative accepted methods and small errors when predicting scoring probabilities in different conditions.", "questions": "**[Q0]** What does employing a risk-based cutoff add to the existing analyst techniques? Having a variety of cut-off thresholds (like all hyperparameters) makes assessment more complex, begging the question: what are the benefits for these additional complexity costs? The empirical results make one useful case about better calibration, but the text would benefit from clarifying the impact and significance to how analysts (or players or coaches) would use this new information.\n\n**[Q1]** What new insights can be obtained from the rankings provided? How are these of interest to those with domain expertise? Practitioners?\n\n**[Q2]** How does RiGIM scale? How much does the model improve as the number of matches observed increases? How effective is the model for evaluating new players (presumably since the player is not a feature of the model cold start problems should be lesser)? How does model training scale in terms of compute resources needed?\n\n**[Q3]** (Table 4) Why might SI do so well at predicting Goals and Game Winning Goals compared to the alternatives? There is a substantial gap between the performance of SI and the runner up for Goals (0.596 vs 0.477) and GWG (0.409 vs 0.266). What is RiGIM failing to capture that SI does?", "limitations": "In terms of limitations, it would help to read some of the authors thoughts on scalability of this technique (mentioned above in the questions). What aspects of risk remain poorly disentangled that are worth further investigation?\n\nThe discussion of the negative impacts of increased player scrutiny is welcome in an era where greater analytics empowers audiences to (often harshly) critique players. I would suggest discussing some of the effects of sports analytics on the behavior of coaches (team decision-makers more broadly) when it comes to evaluating and understanding their players. New analytics can provide insight, but there is always the risk of Campbell's Law and increasing pressure on players to conform to metrics over good behavior (not new in the sense that scoring behavior is already tracked, but now with new measures).", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# strengths\n\nThe method improves over other methods. The results for predict scoring chances are particularly encouraging. The correlational results are more mixed, though the family of proposed methods (*-RiGIM) as a whole obtain competitive results. The bootstrap repetitions in the appendix are convincing that the results are not mere sampling artifacts (though would be welcome in the main text).\n\nThe paper's originality lies in quantifying risk taking (seeking) behavior from offline and potentially sparse data. The core techniques are not particularly novel (distributional RL & density estimation), but they are clearly explained and integrated into a relevant application context. Many analytics practitioners would benefit from understanding these methods and considering the use of uncertainty estimates in their algorithms.\n\nThe exposition is clear and generally easy to follow. Some of the proofs require further background knowledge (not surprising) and the dataset was not clearly explained in the main text. These are minor flaws and less important than conveying the core approach and goals.\n\nThe method shows promising generality. Evaluations on both ice hockey and soccer are favorable, suggesting the decomposition can impact a variety of sports analytic fields (among other domains). This is a welcome advance given the increasing adoption of analytics in many sports and the diversity of existing practices for making human judgments of player performance. Generalization to other offline RL tasks remains an open question, but suggests a broader potential audience at NeurIPS.\n\nThe evaluations demonstrate rigor in handling real-world offline data. For example, splitting time periods and ensuring test data comes later, as would occur when deploying this approach in real world applications.\n\n\n# weaknesses\n\nThe RiGIM model results (Figure 5) are ambiguous. The error bars for ablations heavily overlap the full model, suggesting that the simpler Na-RiGIM may be sufficient in most contexts. The text would benefit from addressing the differences in performance and why they may (not) obtain in this case.\n\nLimited technical novelty. Distributional Bellman and the feature density estimator are both adapted from prior work. The novelty lies more in application than new major results. This is a minor weakness: combining the methods is not trivial.\n\nIt is not clear why risk-sensitive evaluation is valuable to practitioners. Given the post-hoc nature of the analysis, risk quantification will not help balancing exploration and exploitation. This opens the question as to how significant the results are to practitioners. The text would benefit from explicitly addressing this point (see question below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Privacy and Security (e.g., consent)"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept