Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Contribution:
This paper proposes PassNet an architecture designed for soccer pass analytics. PassNet approach is similar to UNet, having a downsampling and upsampling modules with a set of skip-connection between the two modules. To train their model, authors apply the log-loss at the location of the passing event.  Authors evaluate their approach on a soccer analytic dataset, where they demonstrate improvement over prior works relying on hand-crafted features.

Comment:
Contribution of the paper appears a bit incremental to me.  It seems that the paper is a direct application of Unet type of architecture on the specific problem of pass analytic. It would be nice to explicit what is the main contribution of the paper from the representation learning point of view.

In addition, some of the design choice of PassNet could be better justified. It would be nice to run an ablation study to show the importance of the different architectural component (conv2d prediction layers, backpropagating using only one outputs, skip-connection...)

Authors claim that their approach is an "extreme case of weakly supervised learning". I tend to disagree with the assessment. I understand that they only backprop the loss computed at one specific location of the output, but this a choice on the architectural part and not on the labelling. Training PassNet requires fine-grained label as it needs to know both label value and localization. Training the model without the use of label localization would be more akin to weakly-supervised learning.

Authors evaluate their approach on only one dataset.  It would be nice to extend the empirical results to other datasets/sports to ensure the robustness of the conclusions.  

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The aim of the system presented in this paper is to produce a 2D map of probabilities showing the chance of successful completion of a soccer pass to all locations on the field, given coordinate locations of players and the ball sampled over time.  A network based on a fully-convolutional style semantic segmentation network is applied to a 2D, 8-channel game state representation, with final sigmoid layer to predict a pass success indicator at each location.  The system is trained using manually annotated pass success and destinations, which corresponds to a label on the destination point; locations other than the labeled point are not trained (treated as incomplete/"don't-care" training targets).  Evaluation is performed using both log loss and probability calibration measure, to measure effectiveness of predictions as well as how well they calibrate to correspond to probabilities in the sense defined by the measurement.

This is a fun application and it appears the system is effective at a basic level.  However, I think both the theory/explanation and experiments leave a fair bit of uncertainty as to what the probabilities mean and how to interpret them, including conflation of success probability and the model's certainty in its estimate.

In particular, it is unknown to what degree the values output by the model can be interpreted as a predictive probability of pass completion for anywhere on the field.  If one location says 0.8 and the other 0.5, does this mean that if the player were to actually pass to the 0.5 location, the chance of success is actually lower?  Or does it mean that the model is less confident or that this location was under-trained for this state?  The ECE measurement in conjunction with loss error doesn't quite address this:  although a good verification of calibration in its own sense, ECE simply confirms that for cases where the model predicts 0.5, there is success 0.5 of the time for locations that *exist in the test set*, which is sampled according to player action.  To verify the probability maps at all locations, one would need to be able to measure *any* point in the field, not just those already selected by the players' actions.  I think discussion and attempt to measure this is fairly important, as one of the intended applications in the motivation is analyzing what might have happened had a player selected a different destination point.

Unfortunately, knowing the true outcomes at arbitrary locations the players didn't pass to is impossible, so addressing this issue is not straightforward, and unclear to me how it might be done.  A possible suggestion, is to use the destination selection predictions that the paper also mentions can be found using softmax instead of sigmoid.  Although this does not entirely eliminate the issue (these predictions themselves may conflate model confidence with actual selection probability), these maps would likely provide a good indication of player selections.  Thus they might be used to sample or reweight the test set, so that unlikely destination points are sampled more, to try to get a more uniform sample.

Even with this issue, though, I feel that this is an interesting application and system that seems reasonable in its current state, if with important caveats.  Thus, I'd lean towards accepting.  However, I'd encourage the authors to discuss these differences and issues of output interpretation, and to try addressing if possible.


Additional questions and comments:

* The train/val/test split appears to be uniform random by pass event.  It would be interesting to hold out all events for one or more teams, or holding out full games, to measure generalizability to these cases.

* Are the destinations for unsuccessful passes the intercepted location, or an estimated intended location?  Does this difference affect the completion prediction at the intended location (beyond the interception location)?



review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper presents a deep convolutional neural network for estimating, at the given the moment, for any given position on the field, the probability that a pass to that position is successful. It also shows results regarding the probability that a pass to a particular position is attempted. The paper is well-written and the figures and videos do a very nice job of communicating the important ideas. The related work appears to be extensive and the description of the design choices of the architecture and the training procedure is clear and thorough. Based on the results in Table 1, the proposed network appears to compare very favorably against the baselines.

One thing I feel would make the paper stronger is the inclusion of more baselines and an ablation study. Both Logistic Net and Dense2 Net appear to be very lightweight. It is not obvious to me that all of the design choices made in the paper translated to gains in performance. Would a large network perform well without them? You mention that you tried a class-weighting with no sampling approach but that it did perform as well. Why not include these results in the paper?

Small typo: “where achieved by augmentation” -> “were achieved by augmentation”

From the perspective of someone who is not an expert in this area, I think this is a nice paper. It appears to successfully address an interesting problem, it is well-organized, and its solution methodology may have applications to other related problems. My one major criticism is that I feel that there is insufficient information regarding how the design choices affected the performance.
review_title: Official Blind Review #3

Meta-review: The paper proposes PassNet, which is an architecture that produces a 2D map of probability of successful completion of a soccer pass. The architecture has some similarities with UNet and has downsampling and upsampling modules with a set of skip-connections between them.

The reviewers raised several issues:
* Novelty compared to UNET
* Lack of ablation studies
* Uncertainty about what probabilities mean and issues regarding output interpretation.

The authors have tried to address these concerns in their rebuttal and provided additional experiments. They also argue that the application area (sport analytics) of the paper is novel. Even though the application area is interesting and might lead to new problems, this paper did not get enough support from reviewers to justify its acceptance.
Decision: Reject