Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This is a new review, since my previous review was written for the wrong paper. My apologies for the mixup.

The paper introduces a methods based on RL for player evaluation in ice hockey and soccer. The method is called RIGIM
Distributional methods previously used in Atari and Mujoco are now used to evaluate real games.
The main contribution is the introduction of the RIGIM method. The description is quite technical and does not provide intuition how it works, what the short comings are, and how it can be improved.
An evaluation is provided, and a comparison to other methods.  The method has a low correlation to standard measures, but appears to perform somewhat better than other methods. 
The paper does not really provide a discussion on the reasons for this.
My main problems with the paper are the clarity of the message and the clarity of the analysis.
It contains a description of a lot of work done, but what is the problem that is addressed? What is the contribution to the field? Why is it important?

Thanks very much for the message and for the request for clarification.

Thank you for the question. I can emphatize with the authors, and will elaborate on how the paper can be improved and why.

Examples: Section 4 is a technical section, that is missing an intuitive description of their method, and that is missing a high level explanation. Can the authors please add to rheir techjical description a high level explanation of the problems of related work that is adressed by their method. What is different, and why does it work better?

Examples of where the explanations can be improved; In Section 6.1 the authors write: “If we remove SP-CNF or replace it with other uncertainty estimators, most correlations become weaker except for the correlations with the SHP and SHG measures.” Can you please explain or speculate why this is the case?

Section 6, and Table 4 and 5 show correlations that are rather low, yet section 6.2 and section 6.3 fail to address this issue. The authors write: “RiGIM(c) becomes risk-seeking, and thus achieves a higher correlation with success measures. However, the correlations drop when c approaches 0. This observation is consistent with the fact that an overly risk-seeking estimate cannot reflect the real contributions of players.” Why is that so, why can it not reflect the real contributions?

The authors may also consider adding an explicit problem statement and revising their contributions to address questions of the field. As they are, the contributions are a list of work that has been done, failing to address the importance and which insights are contributed to the field.

Overall, I challenge the authors to let their descriptions transcend the description of what they have done, and provide explanations as to the why.
strengths_and_weaknesses: Strengths: 
new method for player evaluation
The method can learn from offline data
Use of distributional RL is interesting

Weaknesses:
Performance correlates poorly with standard measures, although somehat better than other methods
No real insight provided how the method works, and how it can be improved
questions: Please provide a better explanation how the method works, how it builds on other work, how it compares to other work, what the strengths and weaknesses compared to other work are, and please provide better ideas on how to take this further. 
The overall correlations to stnadard measures are quite low, why is that? What can be done to improve?
limitations: No, the limitations are listed very briefly, without real substantiation.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors develop an offline RL model for risk-sensitive agent evaluation and apply it to study player impact in sports games. Unlike previous approaches, their method RiGIM is sensitive to stochasticity in the environment (some situations have lower/higher variation in their outcomes) and can handle the OoD actions that plague Offline RL. Because they are focused on evaluation (and not control), the typical techniques for dealing with OoD actions (lowering action values or constraining the policy) will not work and so the authors develop a scalable density estimator based on normalizing flows. These techniques are evaluated on a large dataset of hockey and soccer games and correlated with traditional measures of impact. 
strengths_and_weaknesses: The paper is clearly written with consistent notation and the problem is well articulated. Both the problem statement and the techniques used to solve it are novel to my knowledge. Furthermore the approach is demonstrated on a large dataset of practical importance with potentially promising results. The main weaknesses are related to the clarity and interpretation of the evaluation (expanded on in the Question section). The evaluation of the results are confusing and it is not clear how this technique should be measured or used. 

Small issues: 

Line 37-39 citation of what algorithms are being contrasted here.
What is the x-axis supposed to represent in (a)-(d) in Figure 1? 

Grammar: "We measure whether RiGIM is sensitive to the risk by its correlation" (Line 287). 

Lines 294-296 are repetitive with 282-283
questions: Can you give some intuitions for the case study results in Table 1/2. The only comment is that there are more defensemen in the higher confidence top 10 -- although this is true empirically -- why is this a prediction of higher confidence? It doesn't seem to clearly correlate with the other metrics shown. Tyler Seguin and Connor McDavid have similar stats and play in the same position so what accounts for them being in the different top 10 lists? The authors mention a couple of qualitative findings e.g., "low confidence favors centers, strong scoring ability) but its not clear what is driving these predictions. How would a layperson use this tool?

For tables 4/5/6, why was c set to 0.5 instead of fit on the validation set? How would one decide on a value for c in practice?

In table 6, many of the bolded numbers are not different at a level of statistical significance. Ideally, the algorithm can be run more times for the camera ready to reduce the uncertainty about performance. 

Why is the correlation with traditional metrics a good way of measuring success? The "success measures" are very simplistic features of the game so it could be that a better correlation with these measures is actually a signal that the algorithm is doing something naive rather than sophisticated. 

Why do traditional metrics better correlate with risk-seeking versions of (smaller values of c)? 

Will the post-hoc calibration techniques developed in this work also apply to the offline RL control setting? Might this be a better approach than constraining the actions that the agent can take? 
limitations: Limitations are addressed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper addresses the problem of evaluating player behavior in games based on the expected value and uncertainty of actions taken, specifically when examining data obtained from live sports (ice hockey and soccer). 

The authors propose a model to decompose games into a series of episodes with inherent stochasticity ("aleatory uncertainty") and limited observational data ("epistemic uncertainty"). The model uses distributional reinforcement learning to address stochastic dynamics. The model also filters samples using a density estimator to address out-of-distribution data and offline data sampling sparsity. Evaluation results show better correlations to game metrics (like scoring goals) when compared to alternative accepted methods and small errors when predicting scoring probabilities in different conditions.
strengths_and_weaknesses: # strengths

The method improves over other methods. The results for predict scoring chances are particularly encouraging. The correlational results are more mixed, though the family of proposed methods (*-RiGIM) as a whole obtain competitive results. The bootstrap repetitions in the appendix are convincing that the results are not mere sampling artifacts (though would be welcome in the main text).

The paper's originality lies in quantifying risk taking (seeking) behavior from offline and potentially sparse data. The core techniques are not particularly novel (distributional RL & density estimation), but they are clearly explained and integrated into a relevant application context. Many analytics practitioners would benefit from understanding these methods and considering the use of uncertainty estimates in their algorithms.

The exposition is clear and generally easy to follow. Some of the proofs require further background knowledge (not surprising) and the dataset was not clearly explained in the main text. These are minor flaws and less important than conveying the core approach and goals.

The method shows promising generality. Evaluations on both ice hockey and soccer are favorable, suggesting the decomposition can impact a variety of sports analytic fields (among other domains). This is a welcome advance given the increasing adoption of analytics in many sports and the diversity of existing practices for making human judgments of player performance. Generalization to other offline RL tasks remains an open question, but suggests a broader potential audience at NeurIPS.

The evaluations demonstrate rigor in handling real-world offline data. For example, splitting time periods and ensuring test data comes later, as would occur when deploying this approach in real world applications.


# weaknesses

The RiGIM model results (Figure 5) are ambiguous. The error bars for ablations heavily overlap the full model, suggesting that the simpler Na-RiGIM may be sufficient in most contexts. The text would benefit from addressing the differences in performance and why they may (not) obtain in this case.

Limited technical novelty. Distributional Bellman and the feature density estimator are both adapted from prior work. The novelty lies more in application than new major results. This is a minor weakness: combining the methods is not trivial.

It is not clear why risk-sensitive evaluation is valuable to practitioners. Given the post-hoc nature of the analysis, risk quantification will not help balancing exploration and exploitation. This opens the question as to how significant the results are to practitioners. The text would benefit from explicitly addressing this point (see question below).
questions: **[Q0]** What does employing a risk-based cutoff add to the existing analyst techniques? Having a variety of cut-off thresholds (like all hyperparameters) makes assessment more complex, begging the question: what are the benefits for these additional complexity costs? The empirical results make one useful case about better calibration, but the text would benefit from clarifying the impact and significance to how analysts (or players or coaches) would use this new information.

**[Q1]** What new insights can be obtained from the rankings provided? How are these of interest to those with domain expertise? Practitioners?

**[Q2]** How does RiGIM scale? How much does the model improve as the number of matches observed increases? How effective is the model for evaluating new players (presumably since the player is not a feature of the model cold start problems should be lesser)? How does model training scale in terms of compute resources needed?

**[Q3]** (Table 4) Why might SI do so well at predicting Goals and Game Winning Goals compared to the alternatives? There is a substantial gap between the performance of SI and the runner up for Goals (0.596 vs 0.477) and GWG (0.409 vs 0.266). What is RiGIM failing to capture that SI does?
limitations: In terms of limitations, it would help to read some of the authors thoughts on scalability of this technique (mentioned above in the questions). What aspects of risk remain poorly disentangled that are worth further investigation?

The discussion of the negative impacts of increased player scrutiny is welcome in an era where greater analytics empowers audiences to (often harshly) critique players. I would suggest discussing some of the effects of sports analytics on the behavior of coaches (team decision-makers more broadly) when it comes to evaluating and understanding their players. New analytics can provide insight, but there is always the risk of Campbell's Law and increasing pressure on players to conform to metrics over good behavior (not new in the sense that scoring behavior is already tracked, but now with new measures).
review_title: null

Meta-review: metareview: The reviews on this paper were mixed. One of the reviewers raised several concerns about the work; although they were valid concerns, our assessment is that the authors have addressed them to a satisfactory degree in their rebuttal.

This is a solid paper describing a systematic, careful, application of techniques from supervised and reinforcement learning to the problem of evaluating sport players taking risk into account. In preparing their final manuscript, we suggest the authors make an effort to explain the application as well as possible to the NeurIPS audience, who may not be familiar with player evaluation in sports game. The interesting discussion between authors and reviewers that followed the initial reviews can be a good source of the type of questions the community  might be interested in. Although the authors have already modified the paper in response to this discussion, we suggest they try to incorporate as much of it as possible to the final version of the paper.
confidence: Less certain
Decision: Accept