Title: RATING CONTINUOUS ACTIONS IN SPATIAL MULTI-AGENT PROBLEMS

Abstract: We study credit assignment problems in spatial multi-agent environments where agents pursue a joint objective. On the example of soccer, we rate the movements of individual players with respect to their potential for staging a successful attack. We propose a purely data-driven approach to simultaneously learn a model of agent movements as well as their ratings via an agent-centric deep reinforcement learning framework. Our model allows for efficient learning and sampling of ratings in the continuous action space. We empirically observe on historic soccer data that the model accurately rates agent movements w.r.t. their relative contribution to the collective goal.

1. INTRODUCTION:

Learning to assess individual behavior in an environment that contains several agents with potentially concurring goals received a lot of attention recently (Lowe et al., 2017; Foerster et al., 2017) , spurred in part by the large success of deep reinforcement learning models in single agent domains.

Figure 1 : Learned ratings of potential movements of the circled red player (green dots, the darker the higher the rating). The red circle denotes the true movement that the player is about to perform.

In this paper, we focus on cooperative multiagent problems in spatial environments on the example of soccer. Figure 1 illustrates a snapshot of a professional soccer game where individual players behave such that the collective is likely to score. We aim to assign a rating to every agent's movement that reflects the quality w.r.t. the collective goal. To do so, we propose to learn possible (and likely) movements of individual players in the unconstrained continuous space.

We translate the spatial multi-agent problem into a Markov Decision Process (MDP) where agents perceive positions and movements of other agents as a state representation that is possibly augmented by environmental variables such as the position of the ball. Given the state, every agent decides on a movement from a continuous action space that aims to support the team in receiving a joint reward. Successful strategies that ground on simulation (Copete et al., 2015; Hausknecht & Stone, 2016) are unfortunately not applicable in our scenario where historic games are the only data source. On the other hand, credit assignments by experts are tedious and only available for a few manually selected scenes per game and usually ground on heuristics (Link et al., 2016; Fernandez & Bornn, 2018) . To remedy the need of experts, we take a purely data-driven approach without external information and focus on learning from observations. On the example of soccer, we show that our complementary approach to expert-based ratings gives meaningful scores that highly correlate with manually created ones by experts.

Technically, we propose an agent-centric deep reinforcement learning approach to rate agent movements in spatio-temporal decision making problems. The real distribution governing each agent's movements in the continuous action space is unknown and can only be observed from examples. We introduce a mixture model that models the unknown distribution of actions, allowing for efficient learning, sampling and evaluation of movements when the examples are limited.

Given the mixture model, we devise a deep reinforcement learning-based approach to learn the values of the mixture components (the ratings of actions). Our approach allows to efficiently evaluate the advantage function of actual movements of the agents in any situation. Both, policy and ratings of movements, are learned simultaneously by jointly optimizing the likelihood and a variant of the Bellman error. The deep model renders the input independent from the roles and positions of the agents and convolutions capture the entire environment from each agent's perspective. Empirically, we show that our approach leads to sensible ratings which are comparable with expert grades on professional soccer data.

2. RELATED WORK:

Cooperative multi-agent scenarios where multiple agents act in a shared environment and pursue a joint objective are usually addressed with multi-agent RL techniques (Tan, 1993; Littman, 2001) . These approaches are often attributed to either team learning aiming at models in joint action spaces or concurrent learning focusing on individual agents (Panait & Luke, 2005) . Claus & Boutilier (1998) demonstrate that the convergence of cooperative concurrent learning could be nearly obtained as in a single-agent setting. Omidshafiei et al. (2017) present a deep learning approach for partially observable multi-agent tasks. Deep RL has been successfully applied in game contexts (Mnih et al., 2015; Lanctot et al., 2017; Silver et al., 2016; 2017) . Moreover, Foerster et al. (2017) focus on learning decentralized policies while a central critic is optimizing independent behaviors.

An actor-critic deep model for an abstraction of RoboCup has been proposed to learn the value of actions (Hausknecht & Stone, 2016) . The action space contains four discrete actions with continuous parameters, one of which captures the magnitude and direction of movement for each agent and thus resembles our movement policies. Agents learn to play the game from scratch in a fully simulated setting, rendering the approach unsuited for problems like team sport that cannot (yet) be simulated. Zhan et al. (2018) and Le et al. (2017b) employ deep imitation learning to model coordinated movements of agents in the problem of basketball and soccer, respectively. Similarly, Le et al. (2017a) use imitation learning to compare the behavior of soccer players to the average performance of either the player herself or a team average. Although the analysis is based on very interesting movement patterns, the average performance does not serve as a reliable evaluation of player movements as it is oblivious of situations and contexts. By contrast, our approach evaluates whether an agent's movement leads to maximizing the actual reward of an episode.

Player rating in soccer is a fairly new topic which aims at quantitatively ranking players instead of subjective measures. The related works consist of defining game indexes (McHale et al., 2012) , space generated by players (Fernandez & Bornn, 2018) , and the location of completed passes (Brooks et al., 2016) . We present a model to learn the value of players movements that gives a fine grained measure of players performance. Other computational approaches of analyzing soccer matches often rely on domain knowledge (Lucey et al., 2014; Bialkowski et al., 2014) or deploy frequency-related criteria to narrow down the candidate space (Haase & Brefeld, 2014) . Scoring opportunities are for instance rated by Spearman (2018) and Link et al. (2016) .

3. LEARNING TO RATE ACTIONS:

3.1 PRELIMINARIES Markov decision processes consist of a state space S, a state-dependent action space M (s), s ∈ S, a reward function r(s, m, s + ), where s, s + ∈ S, and m ∈ M (s), and a state transition distribution p(s + |s, m). We denote the next state and next action by s + and m + , respectively. A stochastic policy π(m|s) describes the distribution of actions m ∈ M (s) that agents in state s will decide for.

In our multi-agent setting, state and action spaces are jointly defined for a set of n agents. A state s ∈ S describes the positions of the agents, {f 1 , ..., f n }, augmented by some other features {g 1 , ..., g n } such as their direction of movement or team ID. Additionally, s encodes the environmental features, e.g., the position and speed of the ball, as well as the location of the goals. An action m ∈ M (s) is defined by the set of movements m = {m 1 , ..., m n } of the agents between two consecutive states s and s + , resulting in a continuous action space.

A soccer game is split into episodes of uninterrupted ball possession phases of a team, hence rendering episodes of variable length. A reward is granted at the last state transition of every episode. We assume a positive reward if the attacking team carries the ball into the dangerous zone of the opponent while retaining ball possession, and zero otherwise. However, note that any other definition of a positive reward may be used as well.

State transition probabilities, p(s + |s, m), are partly deterministic since an agent's position is the sum of previous positions and the performed movements, f +i = f i +m i . Nevertheless, the movement of the ball is a part of the environment and is modeled as random movement. Therefore, the movement distribution is governed by a stochastic policy π(m|s). Although the quantity is unknown, samples from π(m|s) are contained in the episodes. We compute the ratings of the movements using the Bellman equation of the form

Q(s, m) = E s + ∼p r(s, m, s + ) + γE m + ∼π(•|s + ) Q(s + , m + ) .

(1)

The value function recursively computes the cumulative discounted reward until the end of an episode where γ ∈ (0, 1] is a discount factor. In the remainder, we use r instead of r(s, m, s + ) for simplicity in the notation. Equation 1gives the value of the collective movements in the current state, while we are interested in valuating the movements of individual agents m i . Hence, we define

Q(s, m i ) = m: mi =m i π( m|s) π(m i |s) Q(s, m)d m = m: mi =m i p( m|s, m i )Q(s, m)d m (2)

as the value of movement m i of agent i in state s, where π(m i |s) = m: mi =m i π( m|s)d m. In other words, the value of a single agent's movement is the expectation over all collective movements in which that agent performs that movement.

Finally, when rating an agent's behavior, we generally focus on the relative value of a movement compared to the values of alternative movements the agent could perform in the same state. The relative gain of an action is often called the advantage of that action and is defined as

EQUATION

Note that the integral on the right side, that is, the state value function V (s) = mi π( mi |s) Q(s, mi )d mi , is the same for all agents.

3.2. THE OBJECTIVE FUNCTION:

A general approach to learning the value function Q is to interpret the Bellman equation (Equation 1) as an optimality criterion by minimizing the temporal differences. We only have access to a finite amount of historic data consisting of episodes of state, movement, and reward, i.e., (s, m, r) ∈ {(s t , m t , r t )} te t=1 , where t e is the length of an episode. The Bellman objective is given by Ω Q,

EQUATION

where every (s, m i ) is a sample of the policy π and thus Ω Q is an objective for an on-policy policy evaluation from a fixed data set.

A standard approach to learning Q is using a purely sample based method which approximates the integral with a single observed sample which leads to the following loss function

Ω QB = (s,m,r,s + ,m + ) i Q(s, m i ) -r + γ Q(s + , m +i ) 2 .

(5)

However, the number of training examples is limited compared to the complexity of the continuous action space for optimizing such objective. Additionally, we are interested in sampling realistic movements that a player can actually perform. For instance, while in theory the value of running toward goal with a speed of 45 km/h is very high, practically, no player will be able to achieve it. Hence, in order to find the best possible movement of a player in a given state, the model should only be queried for possible actions. This problem arises since the continuous action space is essentially unconstrained and without additional information on the boundaries of player's physical capabilities.

Even with this information, enforcing those bounds can be hard as investigated by Hausknecht & Stone (2016) who proposed to adjust gradients accordingly during learning. Another approach is to discretize the space into a predefined grid (Zheng et al., 2016) .

Instead, we propose a model-based approach to learn the movement policy of the agents that imposes no constraints on the space of movements a priori. In the following, we introduce our model-based approach by devising parametric versions of Q(s, m i ) and π(m i |s) that are simultaneously adapted to data by a deep architecture to minimize Ω Q of Equation 4. Note that the integral over all subsequent movements m +i depends on the form of π and might be difficult to compute. Nonetheless, our parametric approach allows to replace the integral by a computationally inexpensive sum which leads to efficient computation of Q, A, and V (see Section 3.4).

The next section presents a general model for π together with an optimization criterion to learn π from the data. Section 3.4 then introduces a model for Q that makes use of π and leads to a second optimization criterion. Finally, Sections 3.5 and 3.6 bring the loose ends together and present a deep architecture that simultaneously optimizes both criteria.

3.3. ESTIMATING POLICY π:

As described in sections 3.1 and 3.2, the policy π(m i |s) should allow for both computationally efficient integration in Equation 4and efficient sampling of possible movements for m i . We therefore model π as a mixture of k two-dimensional Gaussians

EQUATION

where the two-dimensional function ml Ψ ml (s, i) computes the mean of the l-th Gaussian of agent i in state s. Although parameters Ψ ml are shared across agents, the means of the mixture model are computed for each player i individually. The model allows to sample values efficiently at those k means (see also Figure 1 for an example with k = 5).

Similarly, function σ l Ψ σ l (s, i) maps state s to the width parameter of the spherical covariance matrix σ l Ψ σ l (s, i) • I of the l-th Gaussian of agent i. I denotes the two-dimensional identity matrix. Again, parameters Ψ σ l are shared among all agents. The probability πΨ π (l|s, i) that agent i chooses action l in state s is parameterized by Ψ π . The structure of Ψ π , Ψ ml , and Ψ σl are defined by the deep model in Section 3.6. To not clutter notation unnecessarily, we omit the parameters in the remainder whenever possible.

We estimate the parameters of the Gaussian mixture model by minimizing the negative loglikelihood on the samples. For that we need to estimate p(s + |s). Let s M be the part of the state that is defined by movements of agents such that s +M = {f +i ; ∀i}, where f +i = f i + m i , and let s B denote the remaining part. Then it gives p(s + |s) = p(s +B , s +M |s) = p(s +B |s +M , s)p(s +M |s).

Although p(s +B |s +M , s) remains unknown, samples of p(s +B |s +M , s) can be drawn from the data. Therefore, minimizing the negative log-likelihood of p(s + |s) over data samples {(s, s + )} reduces to p(s +M |s) instead. We further assume that movements of agents are independent given the state and rewrite the state transition function in terms of Equation 6 as p(s +M |s) = i π(m i |s).

The negative log-likelihood of the movement policy, Ω L (Ψ π , Ψ ml , Ψ σl ), thus is written as

EQUATION

3.4. LEARNING VALUE FUNCTION Q:

Recall that the discrete set of movements as defined by the means of the mixture model of agent i in state s is given by { m1 (s, i), ..., mk (s, i)}. Let QΨ Q : (s, ml (s, i)) → be a value function that maps those state-movement pairs to a real number with parameters Ψ Q. In order to generalize to all possible movements, i.e., movements that are not contained in { m1 (s, i), ..., mk (s, i)}, the general value function Q computes the expected value

Q(s, m i ) = l p( ml (s, i)|s, m i , σ l (s, i)) Q(s, ml (s, i)),

where the expectation is w.r.t. p( ml (s, i)|s, m i , σ l (s, i)), the posterior for the movement being generated by the l-th component of the movement model. The following proposition shows that the integral in Equation 4 can be substituted with a sum over the mixture components.

Proposition 1. The integral in Equation 4 can be written as

m+i π( m+i |s + ) Q(s + , m+i ) d m+i = l π(l|s + , i) Q(s + , ml (s + , i)).

Proof. (see Appendix A):

Combining this result with Q yields a simplified form of Ω Q that reduces the learning of Q to learning the parametric function Q which is given by

Ω Q(Ψ Q; Ψ π , Ψ ml , Ψ σl ) = (s,m,r,s + ),i l p( ml Ψ ml (s, i)|s, m i , σ l Ψ σl (s, i)) QΨ Q (s, ml Ψ ml (s, i)) (8) -r + γ l πΨ π (l |s + , i) QΨ Q (s + , ml Ψ ml (s + , i)) 2 ,

3.5. PUTTING THINGS TOGETHER:

The optimization criteria for learning π and Q are determined in the last two sections. Equation 4states a model-based learning objective for learning a generic value function Q that depends on movement policy π, but the parametric functions for π and Q was not defined at that point. We then specified π using a Gaussian mixture model with an optimization criterion for adapting its parameters to data. In the last step, this π was used to define a value function Q whose parameters are learned by minimizing Ω Q in Equation 8. Our choice of π allows to replace the integral in Equation 4 by a sum which leads to an efficiently trainable optimization problem.

In addition, since the computation of Q depends on π, the optimality of Q too depends on the parameters of π in the optimization process Ω Q(Ψ Q; Ψ π , Ψ ml , Ψ σl ). Despite the fact that π and Q could be learned in a pipe-lined approach one after another, we propose a deep model that allows to learn all parameters

Ψ = Ψ Q ∪ Ψ π ∪ Ψ ml ∪ Ψ σl in one single optimization problem min Ψ (Ω Q(Ψ Q; Ψ π , Ψ m, Ψ σ ) + Ω L (Ψ π , Ψ m, Ψ σ )).

We benefit from the convolutional model which enables sharing large parts of the individual parameter sets, thus speeding up training as well as prediction.

3.6. DEEP MODEL:

In this section, we present the deep convolutional model that is developed to learn both Q and π.

More detailed information can be found in Appendix B.

Input: Recall that the state consists of the position and direction of movements of all agents and the ball, augmented by additional information. The input to the net is thus represented by a three dimensional 'image-like' tensor I ∈ R 105×68×12 shown in Figure 2 . Hence, slices through all activation tensors are computed relative to the agent's position. For instance, if the size of the output tensor of the j-th convolutional layer is n j × m j × o j and the players position in the input image is x, y, the relevant slice of the input tensor is located at position x • 105/n j , y • 68/m j and has length o j . Slices through the outputs of all convolutional layers are computed and then compiled into a single feature vector that consists of features of different levels of granularity and receptive fields, centered at the agent's position.

Representation vectors of agents have d dimensions with d = 12 + j=1...6 o j .

An agent's feature representation connects to 5k outputs via a fully connected layer where k is the number of components in the Gaussian mixture model defined in Equation 6(cmp. Figure 4 ). Activations are linear for m and Q, whereas π uses an additional softmax layer to ensure valid probability distributions. Only σ has an exponential activation function to ensure positiveness.

4. EMPIRICAL STUDY:

We evaluate the performance of our model on trajectory data from the UEFA Women's Euro 2017 tournament. The dataset consists of six games of the Dutch team from group stage to the final match. Each game comes as a sequence of x and y-coordinates of all players and the ball sampled at 10 frames per second. Every frame is turned into an input tensor as described in Figure 2 .

We extract all episodes of open play were one team continuously retains ball possession without the game being halted. An episode is considered successful if the team carries the ball into an area that extends 2 meters outside the penalty box of the opponent. In case of a successful action, the episode is labeled with a positive reward, otherwise the reward is 0. We report on results of leave-one-gameout cross validations; that is, every result is the mean of a 6-fold cross validation where each fold contains all episodes of exactly one of the six games. Error bars indicate standard error.

4.1. BASELINES:

As mentioned in Section 3.2, a common approach to learn the value function from data is to approximate the integral in Equation 4 with a single observed sample, yielding the objective in Equation 5.

We compare our model to this baseline. However, as mentioned in Section 3.1, when rating players' movements, we are generally interested in the advantage function of a movement to assess its quality. In order to compute the advantage values, we need an estimate of the state value function V (cmp. Equation 3). We consider three approaches to estimate V . The first two methods estimate the state value function from the learned QB , i.e., we approximate the integral V (s) = m π(m|s) QB (s, m)dm with a sum over J sampled movements V (s) ≈ 1/J J j=1 QB (s, m j ). Since the real movement policy π is unknown, we uniformly sample from the space of possible movements1 , which forms the first baseline called Baseline with Random Sampling. The second baseline uses the policy π from our model to sample from the space of movements and is called Baseline with Model.

The third baseline directly estimates V (s) using the same deep architecture as defined in Section 3.6. Instead of predicting values based on an agent's feature representation, a fully connected layer with a single output V (s) is connected to the last convolutional layer. Thus, V and Q share all model parameters except those of the output nodes. V is estimated by minimizing the expected temporal difference error for states min V (s,s + ,r) (V (s) -(r + γV (s + )))

2 . We show that the learned V assigns meaningful values to states in the Appendix C.1.

4.2. MOVEMENT VALUATION:

The ability to rate individual movements allows us to determine how good a player behaves on the soccer pitch. An example of a rating for one player is shown in Figure 1 for k = 5 mixture components. The hypothesis is that the more agents choose good actions, the more successful the team coordination and thus the higher the reward. 2 We validate the hypothesis as follows.

Evaluation Measures: We present metrics for evaluating both Q-values as well as advantage function A. To evaluate Q, we compute the Q-values of the last completed movements of all players for each episode. In our experiments, we define the duration of movements to be 3 seconds, so that the last completed action starts 3 seconds before the end of an episode. The computed Q-values are averaged over all agents to compute a single value for every episode which is considered a prediction of the outcome of an episode. Hence, the higher the average Q, the higher the probability that the attack is successful. We compute the area under the ROC curve (AUC) for each episode to compare predictions with actual rewards. We refer to this measure as Q AUC.

In order to evaluate A, we compute the advantage function (Equation 3) for every observed movement in an episode for all agents. The values are averaged over all agents and time steps to compute a single value for every episode. Analogous to Q AUC, these values are considered a prediction of the outcome of an episode, and are denoted as Advantage AUC in the results.

To give an intuition of choosing the evaluation measures, first recall that Q(s, m i ) is the expected cumulative reward of m i . Therefore, when we consider an agent's last movement in an episode, i.e. the intended movement 3 seconds before the end of the episode, its value reflects the expected outcome of the episode which serves as a suitable predictor of the outcome. On the other hand, advantage function A describes the relative value of an action. The second evaluation measure thus evaluates that the more qualified decisions the agents make during an episode, the higher the likelihood of success would be.

Performance Results: Figure 5 (left, top) shows the average Q AUC for varying numbers of mixture components of π, with a prediction horizon of 3 seconds. The AUC values of our model grow slightly with the complexity of the model and significantly outperform the baseline in predicting Q-values. This result strongly indicates that using our model-based approach proves advantageous when learning Q from limited data. Figure 5 (left, bottom) plots the average Advantage AUC which also increases with the complexity of the model. The third baseline, that uses the learned V , is not shown in the plot due to giving a value of 0.489 in our experiments. The Baseline with Random Sampling performs poorly, nonetheless, the Baseline with Model that uses the same movement model as our model performs comparable to our approach. This highlights the benefit of learning a movement policy in order to asses the relative value of movements, where our model allows to sample efficiently. Even though our model performs only slightly better than the informed baseline, both approaches significantly outperform the baselines without movement models, In an additional evaluation, we compare our computed movement ratings to expert ratings. We average the advantage of the true movements of nine Dutch players (excluding the goalkeeper) who appeared in all six games over all timesteps in the episode, and compute their performance via cross validation. The models employ k = 5 mixture components and the movement duration is two seconds. We compare our ratings with expert rankings based on Space Occupation Gain (SOG) measure introduced by Fernandez & Bornn (2018) as well as scores from a newspaper3 . Table 1 shows that our approach is in line with the two expert views.

4.3. MOVEMENT MODEL:

An interesting application of the learned movement policies is to compute the possible whereabouts and their likelihood for all players at t seconds in the future. The player with the highest likelihood wins a position. The resulting area won by an agent is sometimes called an agents dominant region as she is likely to control this area (Taki & Hasegawa, 2000) . Figure 5 (right) shows the same game situation as in Figure 1 . The green dots visualize k = 90 mixture components of the player emphasized by the red circle; darker green corresponding to higher likelihood. Prediction is performed for t = 3 seconds into the future. The red zone denotes the controlled zone of that player and the two blue zones correspond to the two blue defenders below and right of the player. We show more examples and evaluations of the likelihood of the movement models in Appendix C.2.

5. CONCLUSION:

We presented a purely data-driven approach to learning to rate agents' actions in collective spatial environments. On the example of soccer, we took an MDP-based approach where agents perceive positions and attributes of other players as the state representation and perform their movements from the continuous action space to pursue a joint goal. We proposed an efficient deep architecture to simultaneously optimize a policy which models realistic movements and a value function that rates them. The choice of model facilitates efficient learning and sampling of actions. Empirically, our model successfully captured the contribution of individual movements to the collective goal. To get a better understanding of the quality of the estimated V of the third baseline, figure 6 shows the performance of state valuation in terms of average AUCs. We compute the value of a frame in a positively labeled episode and compare it with all frames of all negatively labeled episodes where the ball has the same distance to the opponent goal. We hence estimate the probability that V assigns

:

We consider all movements with a speed below 30km/h possible.

And in a real-world application, vice versa, situations in which agents frequently choose suboptimal actions could be reinforced in the coaching.

https://www.ad.nl/nederlands-voetbal/rapport-glanzende-cijfers-voor-de-oranje-leeuwinnen a986670a0/

Figures and Tables:

Figure 2: Input representation to the deep convolutional net. Layers: Team red, team blue, ball, speed in x-direction team red, speed in y-direction team red, speed in x ball,..., team has ball possession {+1, -1}, goal position, border.

Figure 3: Feature representation for an agent as defined by the deep model. Activations of all convolutional layers at the position of the agent are concatenated to form feature representations that are fed into the loss net.

Figure 4: Output nodes and loss.

Figure 5: Top left: Average AUCs for Q-evaluation. Bottom left: Average Advantage AUCs. Right: Exemplary movement model prediction and occupation zones (3 seconds into the future).

Figure 8: Predicted movements (top row) and advantages (bottom row) with mixtures using k = 5 (left), k = 15 (center), and k = 90 (right). Darker green marks higher likelihood (top row) and higher advantage values (bottom row), respectively. Predictions are shown for the circled player and the real next position is depicted by an empty circle.

Figure 9: The slowly moving agent on the left has a circular-shaped movement distribution as there is only little time needed for turning in any direction. The running agent on the right (marked by the red circle) realizes an ellipsoidal-shaped movement distribution.

Players ratings

Table content: <table><tr><td>SOG-based</td><td/><td>Our Model</td><td/><td>Newspaper</td><td/></tr><tr><td>1:Sanden</td><td>1.6</td><td>1:Sanden</td><td>0.086</td><td>1:Martens</td><td>8.5</td></tr><tr><td>2:Miedema</td><td>1.5</td><td>2:Miedema</td><td>0.032</td><td>2:Groenen</td><td>8</td></tr><tr><td>3:Martens</td><td>1.3</td><td>3:Donk</td><td>0.005</td><td>3:Sanden</td><td>7.5</td></tr><tr><td>4:Donk</td><td>1.2</td><td>4:Groenen</td><td>0.004</td><td>3:Miedema</td><td>7.5</td></tr><tr><td>5:Groenen</td><td>1.1</td><td>5:Martens</td><td>0.003</td><td>4:Donk</td><td>7</td></tr><tr><td>6:Spitse</td><td>0.85</td><td>6:Dekker</td><td>0.002</td><td>4:Dekker</td><td>7</td></tr><tr><td>7:Van Es</td><td>0.8</td><td>7:Gragt</td><td>-0.015</td><td>4:Gragt</td><td>7</td></tr><tr><td>8:Dekker</td><td>0.69</td><td>8:Spitse</td><td>-0.018</td><td>4:Spitse</td><td>7</td></tr><tr><td>9:Gragt</td><td>0.67</td><td>9:Van Es</td><td>-0.075</td><td>4:Van Es</td><td>7</td></tr></table>

Peer review (iclr2019):
 This paper presents a method for learning the Q function from multi-agent demonstrations, such as trajectories of soccer players in a soccer game. The basic ideas are: -- The policy class pi(m|s) maps states s to actions m probabilistically. In particular, the probability class is a mixture of Gaussians, with the mean/covariance functions and the mixture function all being instantiated via neural nets. The behavior of the individual players are assumed to be conditionally independent given the current state. -- A hand-designed reward function is used, such as the ball being in a "strike zone" of some kind for the team on offense. -- The Q function class is also a neural net. -- The policy pi is learned through maximum likelihood of the state/action pairs of the the pre-collected demonstrations. This is essentially probabilistic behavioral cloning over the policy class. -- The Q function is learned via Q-fitting over the demonstration data, i.e., minimizing squared error on the Bellman residual. This requires the pre-specified reward function, the policy pi, and some details regarding the state transitions (e.g., the non-deterministic transitions of the ball). -- The policy pi and the Q function are trained jointly by adding the two objectives. -- The main empirical evaluation is comparing the quality of the Q function estimation against a naive baseline (random movements) and a non-mixture policy class (i.e., a single mode, rather than a mixture of Gaussians). The results against the non-mixture policy class do not seem to show much difference in performance. -- Some qualitative evaluations are also presented, but it*s unclear what insight one is expected to gain from looking at them. **Clarity** The paper is reasonably well written. Some aspects of the logical flow could be improved, but overall it*s fine. Detailed comments are: -- The proposed approach is essentially imitation learning and not, strictly speaking, reinforcement learning. Yet the work is positioned as reinforcement learning. Perhaps it*s just a difference in norms of terminology usage. -- There are more multi-agent RL & IL works than what was discussed in the related works section. Of course, an exhaustive discussion is not expected, but the writing makes it sound like there hasn*t been much work at all. Examples include: https://arxiv.org/abs/1807.09936 https://arxiv.org/abs/1706.02275 http://www.cs.ox.ac.uk/people/shimon.whiteson/pubs/rashidicml18.pdf -- The authors comment in the related work that team sports cannot yet be simulated. This statement is sort of true, although it*s not clear how true. For instance, the physics engines of many sports video games are very realistic. Moreover, it doesn*t strike me as the most prominent point of contrast with [Hausknecht & Stone 2016]. For instance, a more prominent point of contrast is simply imitation learning vs reinforcement learning. -- It*s not clear what the authors mean specifically when they refer to "average performance" contrasting with [Le et al. 2017b]. Also, desn*t that work also condition on game context and situation? -- Section 4.1 is written in a somewhat disorganized way. The exposition alternates between discussing baselines and the evaluation methodology in a way that is confusing. -- One of the baselines is described but then left to the appendix for analysis. This doesn*t make sense from a narrative perspective. **Originality** From a technical perspective, it*s not clear that there*s much novelty in this approach. All the ingredients are pretty standard, and the ingredients are put together in a standard way. From an applications perspective, there might be some novelty here but it*s not clear. The application of imitation learning to sports data is not new, but also not saturated either. The specific idea of learning a better rating system is interesting, but it*s not clear how fleshed out this application is in the paper (more comments on this below). **Significance** For me, the key question for significance boils down to: "Does this paper change the way people think about doing X?" for the most interesting instantiation of X possible. In this case, it seems X should be computing a rating system for multi-agent spatial multi-agent behavior (as suggested by the title). But the limitations in the model and the experiments make this a questionable proposition. The model class all but ignores the multi-agent aspect (the agents are all independent conditioned on the states). Moreover there was no comparison against a model class that made a worse assumption on how to handle the multi-agent aspect. So it*s not clear how the multi-agent aspect is significant. The evaluation of the rating approach is not convincing. The approach essentially performed as well as a single-mixture baseline. Furthermore, the direction V(s) learning approach is deferred to the appendix, and I*d like to see it included in the main paper. Finally, it would be nice if there was some attempt to compare with a non-MDP based approach. For instance, people directly compute things like P(goal|state). I*d like to see evidence that this style of approach will outperform those approaches, which are the de facto standard in sports analytics right now. **Overall Quality** Based on the above comments, it is my opinion that this paper has too many holes in it to be ready for publication at a venue such as ICLR. As a primarily applications paper, the burden is on the authors to demonstrate reasonably convincing evidence that this line of approach is valuable to the application.