baller2vec: A Multi-Entity Transformer For Multi-Agent Spatiotemporal Modeling
Authors: Michael A. Alcorn
Anh Totti Nguyen
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Representation learning

Abstract: Multi-agent spatiotemporal modeling is a challenging task from both an algorithmic design and computational complexity perspective. Recent work has explored the efficacy of traditional deep sequential models in this domain, but these architectures are slow and cumbersome to train, particularly as model size increases. Further, prior attempts to model interactions between agents across time have limitations, such as imposing an order on the agents, or making assumptions about their relationships. In this paper, we introduce baller2vec, a multi-entity generalization of the standard Transformer that can, with minimal assumptions, simultaneously and efficiently integrate information across entities and time. We test the effectiveness of baller2vec for multi-agent spatiotemporal modeling by training it to perform two different basketball-related tasks: (1) simultaneously forecasting the trajectories of all players on the court and (2) forecasting the trajectory of the ball. Not only does baller2vec learn to perform these tasks well (outperforming a graph recurrent neural network with a similar number of parameters by a wide margin), it also appears to "understand" the game of basketball, encoding idiosyncratic qualities of players in its embeddings, and performing basketball-relevant functions with its attention heads.

baller2vec: A Multi-Entity Transformer For Multi-Agent Spatiotemporal Modeling
Anonymous Author(s)
Affiliation Address
email
Abstract
Multi-agent spatiotemporal modeling a challenging task from both an algorithmic design and computational complexity perspective. Recent work has explored the efficacy of traditional deep sequential models this domain, but these architectures are slow and cumbersome to train, particularly model size increases. Further, prior attempts to model interactions between agents across time have limitations, such as imposing order the agents, or making assumptions about their relation- ships. In this paper, we introduce baller2ved" a multi-entity generalization of the standard Transformer that can, with minimal assumptions, simultaneously and efficiently integrate information across entities and time. We test the effectiveness of baller2vec for multi-agent spatiotemporal modeling by training to perform two different basketball-related tasks: (1) simultaneously modeling the trajectories of all players the court and (2) modeling the trajectory of the ball. Not only does baller2vec learn to perform these tasks well (outperfrmmigg a graph recurrent neural network with similar number of parameters by wide margin), it also appears to "understand" the game of basketball, encoding idiosyncratic qualities players in its embeddings, and performing basketball-relevan functions with its attention heads.
11
13
15
17
Introduction
18
19
Whether it is defender anticipating where the point guard will make pass game of basketball, a marketing professional guessing the next trending topic on a social media platform, or theme park manager forecasting the flow of visitor traffic, humans frequently attempt to predict phenomena arising from processes involving multiple entities interacting through time. When designing learning algorithms to perform such tasks, researchers face two main challenges:
20
21
22
23
24
1. Given that entities lack a natural ordering, how do you effectively model interactions between entities across time?
25
28
2. How do you efficiently learn from the large, high-dimensional inputs inherent to such sequential data?
27
28
Prior work in athlete trajectory modeling, a widely studied application of multi-agent spatiotemporal modeling (MASM; where entities agents moving through space), has attempted model player interactions through "role-alignment" preprocessing steps (i.e., imposing an order on the players)
30
31
or graph neural networks (3), but these approaches may destroy identity information in the former case (see Section 4.2) or limit personalization in the latter case (see Section 5. Recently, researchers have experimented with variational recurrent neural networks (VRNNs) 4 to model
2
32
33
All and code the paper are available at: <anonymizeds.
Submitted to Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

34 the temporal aspects of player trajectory data B 2), but the inherently sequential design of this architecture limits the size of models that can feasibly be trained experiments.
35
36
Transformers [ were designed to circumvent the compurational constraints imposed by other sequential models, and they have achieved state-of-the-art results in wide variety of sequence learning tasks, both in natural language processing (NLP), c.g., GPT-3 [6], and computer vision, c.g., Vision Transformers 77. While Transformers have successfully been applied to static multi-entity data, e.g., graphs [8]. the only published work we are aware of that attempts to model multi-entity sequential data with Transformers uses four different Transformers to separately process information temporally and spatially before merging the Tub-Tranfformer outputs (9]
38
39
40
41
42
In this paper, introduce multi- entity Transformer that, with minimal assumptions, is capable of simultane- ously integrating information across agents and time, which gives pow- erful representational capabilities. We adapt the original Transformer archi- tecture to suit multi-entity sequen- tial data by converting the standard self-attention mask matrix used in NLP tasks into novel self-attention mask tensor. To test the effectiveness of our multi-entity Transformer for MASM, we train it to perform two dif- ferent basketball-related tasks (hence the name baller2vec): (1) simulta- neously modeling the trajectories of all players the court (Task P) and (2) modeling the trajectory of the ball (Task B). Further, convert these tasks into classification problems by binning the Euclidean 83
44
46
47
48
49
Figure I: After solely being trained to model the trajectory of the ball (0) given the locations of the players and the ball on the court through time, self-attention (SA) head in baller2vec learned to anticipate passes. When the ball handler O) is driving towards the basket at t = 4, SA assigns near-zero weights (black) to all players, suggesting 56
50
51
52
53
54
55
no passes will be made. Indeed, the ball handler did not pass and dribbled into the lane weight (white) to a teammate (O), which correctly identifies the recipient of the pass at 13.
57
8). SA then assigns high
59
61
trajectory space, which allows baller2ved to learn complex, multimodal trajectory distributions via strictly maximizing the likelihood of the data (in contrast to variational approaches, which maximize the evidence lower bound and thus require priors over the latent variables). We find that:
65
1. baller2vec is an effective learning algorithm for MASM, obtaining perplexity of 1.64 on Task P (compared to 15.72 when simply using the label distribution from the training set) and 13.44 Task B (vs. 316.05) (Section 4.1) Further, compared to a graph recurrent neural network (GRNN) with similar capacity, baller2vec is 338 times faster and achieves a 10.5% lower average negative log-like likeliool (NLL) on Task P (Section 4.1).
67
69
71
2. baller2vec demonstrably integrates information across both agents and time to achieve these results, evidenced by ablation experiments (Section 4.2
73
3. The identity embeddings learned by baller2vec capture idiosyncratic qualities of players, indicative of the model's deep personalization capabilities (Section 4.3)
75
4. baller2vec's trajectory bin distributions depend on both the historical and current context (Section 4.4). and several attention heads appear to perform different basketball-relevant functions (Figure Section 4.5). which suggests the model learned to "understand" the sport.
76
77
78
2
Methods
79
80
2.1
Multi-entity sequences
81
Let = 21, B} be set indexing B entities and P (p1, (P1.P2.... P2.... . C be the K entities
82 involved in particular sequence. Further, let Z, = { 21,2...... 2, be an unordered set of K feature vectors such that 21,x is the feature vector time step t for entity Pk- Z (Z1, Zz. ... Zz) is
84 thus an ordered sequence of sets of feature vectors over T time steps. When K = 1, is a sequence 85 of individual feature vectors, which is the underlying data structure for many NLP problems.
86 now consider two different tasks: (1) sequential entity labeling, where each entity has its own label at each time step (which is conceptually similar to word-level language modeling), and (2) sequential

---

labeling, where each time step has single label (see Figure 3) For (1), let V = (V1, V2...., Vr) be 89 a sequence of sets of labels corresponding to such that label at time step / for the entity indexed by k. For (2), let W = (w1, w2,.... (eT) be sequence of 91 labels corresponding to where Wy the label time step The goal is then learn function
88
{ur,1 0t,2 . Vt,K and vr the
that maps set of entities and their time-dependent feature vectors E to probability distribution over either (1) the entities' time-dependent labels V or (2) the sequence of labels W.
98
94
2.2
Multi-agent spatiotemporal modeling
n..
Agent starting point
95
In the MASM setting, P is a set of K different agents and
96
C = ((11,1, (Z Vr.2).... ( Vt.k) {)] is an un- ordered set of K coordinate pairs such that ( Mt,k) are the coordinates for agent Pk time step The ordered sequence of sets of coordinates C = (C1, C2...., Cr), together with P, thus defines the trajectories for the K agents over T time steps. We then define 2, as:
72
97
96
99
100
Agent ending point
2 = g(e(()), Kt, Yr,,,, 11,,)),, where is a multilayer 102
An example of binned tra- jectory. The agent's starting position is at the center of the grid, and the cell containing the agent's ' ending position is used the label (of which there n² possibilities).
Figure 2:
an agent embedding layer, and h, a vector of optional contextual features for agent p& at time step t. The trajectory for agent Pk time step is defined as (Zz+1.k - ke Mt+ - k), Similar to Zheng et al. 10). to fully capture the multimodal nature of the trajectory distributions, we binned the 2D Euclidean space into X grid (Figure 2 and treated the problem responding sequence sets trajectory labels (i.e., 0t = Bin ATi,k, Ay,,)), so 02 is an integer
perceptron (MLP),
104
106
108
classification task. Therefore, 2 has
110
from one to n²²), and the loss for each sample in Task P is: L = -=E K (2(f(Z),, [ve,),) where f()),, [Va, is the probability assigned to the trajectory label for agent Pk, at time step by f;
112
Le., the loss is the NLL of the data according the model.
For Task B, the loss for each sample is: L Ee= In(f(z)) [wx]), where f(Z)[[w) is the probability assigned to the trajectory label for the ball at time step by f. and the labels correspond to binned 3D Euclidean space (i.e., w, Bin Ax, Ayr, Azt), so wy is an integer from one to n²").
115
Ui Bin/ A Atca)
Binl An,, Aye Ane)
Vi
V
1
M1g
Fy
Transformer
Transformer
Z
Zx
7/
Zh
Zx
O= Fe.t (((e(pp)) Be,b. he,.)))))
Figure 3: An overview of our multi-entity Transformer, baller2vec. Each time step consists of an unordered set Z, of entity feature vectors (colored circles) as the input, with either (left) corresponding set V; of entity labels (colored diamonds) or (right) a single label (gray triangle) as the target. Matching colored circles/diamonds across time steps correspond to the same entity. In our experiments, each entity feature vector 2t,x is produced by an MLP g that takes player's identity embedding ((pk), ), raw court coordinates (Jr,,;; Vt.,:,, and binary variable indicating the player's frontcourt h input. Each entity label 02,4 is an integer indexing the trajectory bin derived from the player's raw trajectory, while each wi an integer indexing the ball's trajectory bin.
2.3 The multi-entity Transformer
3

---

118 We now describe our multi-entity Transformer, baller2vec (Figure B). For NLP tasks, the Transformer self- attention mask M takes the form of a
1=2
119
121
122
T x T matrix (Figure 9 where T is the length of the sequence. The ele- ment at Mt thus indicates whether or not the model can "look" at time step when processing time step t1. Here, we generalize the stan- dard Transformer to the multi-entity setting by employing a T x K x T x K mask tensor where element Mr, ky d2 indicates whether or not the model can "look" at agent Pkg
123
125
Figure 4: Left: the standard self- mask matrix M. The element at M,,,, indicates whether or not the model can "look" at time step l2 when processing time step 4, Right: the matrix form of our multi-entity self-attention mask tensor. In tensor form, element Mt f2.82 indicates whether or not the model can "look" at agent Pka at time step 12 when processing agent Pe, at time step t1. In matrix form, this corresponds to element Mr, K t2 K2 when using zero- based indexing. The M shown here is for a static, fully connected graph, but other, potentially evolving network structures can be encoded in the attention mask tensor.
127
129
130
131
132
133
time step t2 when processing agent ph. at time step 41. Here, we mask all elements where l2 > i and leave all remaining elements unmasked, i.e., baller2vec is a "causal" model.
134
135
136
137
In practice, to be compatible with Transformer implementations in major deep learning libraries, we reshape M into a TK x TK matrix (Figure 4 and the input to the Transformer is a matrix with shape TK F where F is the dimension of each 2t,k- Irie et al. [11] observed that positional encoding (5] is not only unnecessary, but detrimental for Transformers that use a causal attention mask, we do not use positional encoding with baller2vec. The remaining computations are identical to the standard Transformer (see code).
139
140
141
142
143
144
3
Experiments
145
3.1
Dataset
146
We trained baller2vec on publicly available dataset of player and ball trajectories recorded from
147
631 National Basketball Association (NBA) games from the 2015-2016 season All 30 NBA teams and 450 different players were represented. Because transition sequences are strategically important part of basketball, unlike prior work, e.g., Felsen et al. [.. Yeh et al. [3], Zhan al. (2). we did not terminate sequences on change of possession, nor did we constrain ourselves to a fixed subset of
148
149
151
sequences. Instead, each training sample was generated on the fly by first randomly sampling a game, and then randomly sampling a starting time from that game. The following four seconds of data were downsampled to 5 Hz from the original 25 Hz and used as the input.
152
153
154
Because we did not terminate sequences on a change of possession, we could not normalize the direction of the court as was done in prior work N 3 2. Instead, for each sampled sequence, we randomly (with probability of 0.5) rotated the court 180 (because the court's direction is arbitrary), doubling the size of the dataset. We used a training'validationneest split of 569/30/32
155
156
157
158
games, respectively (i.e., 5% of the games were used for testing, and 5% of the remaining 95% of games were used for validation). As result, we had access to ~82 million different (albeit overlapping) training sequences (569 games 4 periods per game x 12 minutes per period x 60 seconds per minute x 25 Hz. x rotations), ~800x the number of sequences used in prior work. For both the validation and test sets, 1,,00 different, non-ovvrrapping sequences were selected for evaluation by dividing each game into [ 1.0000 non-overlapping chunks (where N is the number of games), and using the starting four seconds from each chunk as the evaluation sequence.
159
161
163
164
185
3.2
Model
186
We trained separate models for Task P and Task B. For all experiments, we used a single Transformer architecture that was nearly identical to the original model described in Vaswani al. (5), with dmalel = 512 (the dimension of the input and output of each Transformer layer), eight attention heads,
187
168
http:://ggthuu..cc///noooo///// Mlaver--entttt
Movements

---

189
dr = 2048 (the dimension of the inner feedforward layers), and six layers, although we did not use dropout. For both Task and Task B, the players and the ball were included in the input, and both the players and the ball were embedded to 20-dimensional vectors. The input features for each player consisted of his identity, his (ar,)) 3) coordinates on the court at each time step in the sequence, and a binary variable indicating the side of his frontcourt (i.e., the direction of his team's hoop) The input features for the ball were its (z, y, 2) coordinates at each time step.
170
172
178
174
175
The input features for the players and the ball were processed by separate, three-layer MLPs before being fed into the Transformer. Each MLP had 128, 256, and 512 nodes in its three layers, respectively, and ReLU nonlinearity following each of the first two layers, For classification, single linear layer was applied to the Transformer output followed by a softmax. For players, we binned an 11 ft 11 2D Euclidean trajectory space into an 11 x 11 grid of 1 ft x 1 ft squares for total of 121 player trajectory labels. Similarly, for the ball, we binned a 19 ft x 19 ft x 19 ft 3D Euclidean trajectory space into a 19 x 19 x 19 grid of 1 ft x 1 ft x 1 ft cubes for total of 6,859 ball trajectory labels.
176
177
178
179
180
182
We used the Adam optimizer [12] with an initial learning rate of 10 31 0.9, 32 0.999, and = 10-9 to update the model's parameters, of which there were ~ million for Task P/Task B, respectively. The learning rate was reduced to 10-7 7 after 20 consecutive epochs of the validation loss not improving. Models were implemented in PyTorch and trained on a single NVIDIA GTX 1080 Ti GPU for seven days (~~50 epochs) where each epoch consisted of 20,000 training samples, and the validation set was used for early stopping.
183
184
185
186
187
188
3.3
Baselines
As our naive baseline, we used the marginal distribution of the trajectory bins from the training set for all predictions. For our strong baseline, we implemented a baller2vec- like graph recurrent neural network (GRNN) and trained it Task P (code is available in the baller2ver repos- it Specifically, at each time step, the player and ball inputs were first processed using MLPs as in baller2ves and these inputs were then fed into a graph neural net- work (GNN) similar to Ych et al. [3] The node and edge functions of the GNN were each a Transformer- like feedforward network (TFF), i.e., TFF(>) = LN(x + W2ReLU + b,) + b2), where LN is Layer Normal- ization [3] W, and W2 are weight matrices, by and b2 are bias vectors, and ReLU is the rectifier activation function. For our RNN, we used a gated recurrent unit (GRU) RNN 14 in which we replaced each of the six weight matrices of the GRU with a TFR Fach TFF had the same dimensions as the Transformer layers used in baller2vec. Our GRNN had
Table I: on the test set when using baller2vec vs. the marginal distribution of the tra- jectory bins in the training set ("Train")) for all predictions. baller2vec consid- erably reduces the uncertainty over the trajectory bins.
The perplexity per trajectory bin
190
192
183
194
195
196
baller2vec
Train
197
198
Task P
1.64
15.72
199
Task B
13.44
316.05
200
201
203
204
205
~ 18M parameters, which is comparable to the 19M in baller2vec. We also trained our GRNN
for seven days (~5175 epochs).
207
3.4
Ablation studies
208
To assess the impacts of the multi- design and player embeddings of baller2vec on model performance, we trained three variations of our Task P model using: (1) one player in the input
209
210
without player identity, (2) all 10 players in the input without player identity, and (3) all 10 players in the input with player identity. In experiments where player identity was not used, single generic
211
212
player embedding was used in place of the player identity embeddings. We also trained two variations
213
of our Task B model: one with player identity and one without. Lastly, to determine the extent to which baller2vec uses historical information in its predictions, we compared the performance of
214
our best Task P model on the full sequence test set with its performance on the test set when only predicting the trajectories for the first frame (i.e., we applied the same model to only the first frames of the test set).
216
We did not include team identity as an input variable because teams are collections of players and a coach, and coaches did not vary in the dataset because we only had access half of season of data; however, with additional seasons of data, we would include the coach as an input variable. "We chose to implement our own strong baseline because baller2vec has far more parameters than models from prior work (e.g.. 70x Felsen et al. (I].
5

---

4
Results
218
219
4.1
baller2vec is an effective learning algorithm for multi-agent spatiotemporal modeling.
220
The average NLL on the test set for our best Task P model was 0.492, while the average NLL for our best Task B model was 2.598. In NLP, model performance is often expressed in terms of the perplexity per word, which, intuitively, is the number of faces on a fair die that has the same amount of uncertainty as the model per word (i.e., a uniform distribution over M labels has a per- plexity of M, model with per word perplexity of six has the same
Table 2: The average NLL (lower is better) on the Task P test set and seconds per training epoch (SPE) for baller2vec (b2v) and our GRNN. baller2vec trains ~3.8 times faster per epoch compared to our GRNN, and baller2ves outper- formed our GRNN by 10.5% when given the same amount of training time. Even when only allowed to train for half ("0.5x"") and quarter ("0.25x") as long as our GRNN, baller2vec outperformed our GRNN by 9.1% and 1.5%, respectively..
222
224
226
228
230
b2v
b2v (0.5x)
b2v (0.25x)
GRNN
231
NLL
0.492
0.499
0.541
0.549
232
233
average uncertainty as rolling a fair six-sided die). In our case, we con-
SPE
~00
~900
~900
~3,400
234
sider the perplexity per trajectory bin, defined as: PP = extx Σ. ΣT_ Σ - (n(p(vw...))) where N is the number of sequences. Our best Task P model achieved a PP of 1.64, i.e., baller2vec was, average, as uncertain as rolling a 1.64-sided fair die (better than a coin flip) when predicting player trajectory bins (Table D. For comparison, when using the distribution of the player trajectory bins in the training set the predicted probabilities, the PP on the test set was 15.72. Our best Task model achieved a PP of 13.44 (compared to 316.05 when using the training set distribution).
235
236
238
240
241
Compared to our GRNN, baller2vec was 3.88 times faster and had 10.5% lower average NLL 242
when given an equal amount of training time (Table 2 Even when only given half as much training time as our GRNN, baller2vec had a 9.1 % lower average NLL.
243
244
4.2
baller2vec uses information about all players on the court through time, in addition to player identity, to model spatiotemporal dynamics.
246
Results for our ablation experiments can be seen in Table 3 Including all 10 players in the input dra- matically improved the performance of our Task P
Table The average NLL on the test set for each of the models in our ablation experi- ments (lower better). For Task P, using all 10 players improved model performance by 18.0%, while using player identity improved model performance by an additional 4.4%. For Task B, using player identity improved model performance by 2.7%. 1/10 indicates whether one or 10 players were used as input, respectively, while I/NI indicates whether or not player identity was used, respectively.
247
248
249
model by 18.0% vs. only including a single player. Including player identity improved the model's 's per-
250
251
formance further 4.4%. This stands in contrast to
252
Felsen et al. [1] where the inclusion of player identity led to slightly worse model performance; a coun- terintuitive result given the range of skills among NBA players, but possibly a side effect of their role- ali aligneent procedure. Additionally, when replacing the players each test set sequence with random players, the performance of our best Task P model de- teriorated by 6.2% from 0.492 to 0.522. Interestingly, including player identity only improved our Task B model's s performance by 2.7%. Lastly, our best Task the full sequence test set (0. (0.492) was 70.6% better than its performance on the single frame test set (1.67), i.e., baller2vec is clearly using historical information to model the spatiotemporal dynamics of basketball.
253
254
255
257
Task
I-NI
10-NI
10-1
259
Task P Task B
0.628
0.515
0.492
N/A
2.670
2.598
261
262
P model's is performance
263
285
4.3
baller2vec's learned player embeddings encode individual attributes.
286
Neural language models are widely known for their ability to encode semantic relationships between words and phrases as geometric relationships between embeddings see, e.g., Mikolov et al. [16 Le and Mikolov [18], Sutskever al. [9). Alcorn (20) observed a similar phenomenon in a
268
17,,
baseball setting, where batters and pitchers with similar skills were found next to each other in the

---

Pooer Fanaid
Saheraergg Dooks
Small Fanord
.12
Point Sauid
Bowh Leokely
Vieebookk
-1.
Dit Nowttzkk
BRoon Damrs
Ametokkarmmrg
Stephicn Carry
115
5Cho Pail
Figure 5: As can be seen in this 2D UMAP of the player embeddings, by exclusively learning to predict the trajectory of the ball, baller2vec was able to infer idiosyncratic player attributes. The left-hand side of the plot contains tall post players (2, 2) e.g., Serge Ibaka, while the right-hand side of the plot contains shorter shooting guards (1²) and point guards (+), e.g., Stephen Curry. The connecting transition region contains forwards (I), O) and other "hybrid" players, i.e., individuals possessing both guard and post skills, e.g.. LeBron James. Further, players with similar defensive abilities, measured here by the cube root of the players' blocks per minute in the 2015-2016 season 15 cluster together.
270
embedding space learned by a neural network trained to predict the outcome of an at-bat. A 2D
271
UMAP [21] of the player embeddings learned by baller2vec for Task B can be seen in Figure 5 Like (batter pittcher) 2vec (20), baller2vec seems encode skills and physical attributes in its player embeddings,
272
273
Querying the nearest neighbors for individ- ual players reveals further insights about the baller2vec embeddings. For example, the nearest neighbor for Russell Westbrook, an ex- tremely athletic 6'3" point guard, is Derrick Rose, a 6'2" point guard also known for his athleticism. Amusingly, the nearest neighbor for Pau Gasol, a 7' center with a respectable shooting range, is his younger brother Mare Gasol, a 6'11" center, also with respectable shooting range.
13
275
278
277
278
279
280
281
282
284
11.8
f1.8
16.8
fa,
V13,8
f13,8
Figure 6: baller2vec's trajectory predicted tra- jectory bin distributions affected by both the historical and current context. At L = 1, fairly uncertain about the target player's (A; k = 8) trajectory (left grid and dotted red line; the blue-bordered center cell is the "sta- tionary" trajectory), with most of the probability mass divided between trajectories moving towards the ball handler's 's sideline (right grid; black = 1.0; white = 0.0). After observing portion of the sequence (t 6), baller2vec becomes very cer- tain about the target player's trajectory (fo,s), but when the player reaches a decision point (1 13), baller2vec becomes split between trajectories
285
4.4 baller2vec's predicted trajectory bin distributions depend on both the historical and current context.
286
baller2vec
288
Because baller2vec explicitly models the dis- tribution of the player trajectories (unlike varia- tional methods), we can easily visualize how its predicted trajectory bin distributions shift in different situations. As can be seen in Fig- ure 6 baller2vec's predicted trajectory bin distributions depend on both the historical and current context. When provided with limited historical information, baller2vec tends to be less certain about where the players might go. baller2ved also tends to be more certain when predicting trajectory bins at "easy" mo- ments (e.g., player moving into open space) vs. "hard" moments (e.g., an offensive player choos- ing which direction to move around defender).
289
290
282
293
295
296
297
(staying still or moving towards the top of the key). Additional examples can found in Fig- ball, Ll offense, A defense, and
299
ST
fek
f(2) L
301
7

---

303
4.5
Attention heads in baller2vec appear to perform basketball-releeant functions.
304
One intriguing property of the attention mecha-
305
nism 25 is how, when visualized, the atten-
306
tion weights often seem to reveal how model
307
is "thinking". For example, Vaswani et al. [S]
308
discovered examples of attention heads in their Transformer that appear to be performing var-
309
310
ious language understanding subtasks, such as anaphora resolution. As can be seen in Figure
Figure 7: The attention outputs from baller2vec suggest learned basketball-relevatt functions. Left: attention head 2-7 (layer-head) appears to focus on teammates of the ball handler (mO). Mid- dle and right: attention head 6-2 seems predict (middle; A) who the ball handler will be in a future frame (right). Players are shaded according the sum of the attention weights assigned to the play- through time with reference to the ball in the current frame (recall that each player occurs mul- tiple times in the input). Higher attention weights are lighter. For both of these attention heads, the sum of the attention weights assigned to the ball through time was small (0.01 for both the left and middle frames where the maximum is 1.00). Ad- ditional examples can be found in Figures S2 and
312
7 some of the attention heads in baller2vec seem to be performing basketball understand- ing subtasks, such as keeping track of the ball handler's teammates, and anticipating who the ball handler will pass to, which, intuitively, help with our task of predicting the ball's trajectory.
314
316
317
5
Related Work
319
5.1
Trajectory modeling in sports
320
There is a rich literature on MASM, particularly in the context of sports, e.g., Kim et al. (26]. Zheng et al. [10.. Le et al.
321
27 28 Qi et al.
323
29). Zhan et al. 30 Most relevant to our work
S
324
is Yeh et al. 3] who used variational recurrent neural network combined with a graph neural
325
network to forecast trajectories in a multi-agent setting. Like their approach, our model is permutation cquivariant with regard to the ordering of the agents; however, we use a multi-head attention
327
mechanism to achieve this permutation equivariance while the permutation equivariance in Yeh et al. 3 is provided by the graph neural network. Specifically, Yeh et al. [3] define: v - e ei, = fe ([v., yj. (,y and - v o, fo (ExExx [er,,,(]] where v; is the initial state of agent tu is an embedding for the edge between agents i and j. e is the representation for edge (i, Mr the neighborhood for agent i, t; is a node embedding for agent i, o, is the output state for agent i, and fe and fo are deep neural networks.
329
331
332
333
334
Assuming each individual player is a different "type" in fe (i.e., attempting to maximize the level of personalization) would require 450² = 202,500 (i.e., B²) different t., edge embeddings, many of which would never be used during training and thus inevitably lead to poor out-of-sample performance. Reducing the number of type embeddings requires making assumptions about the nature of the relationships between nodes. By using a multi-head attention mechanism, baller2ver learns to integrate information about different agents in highly flexible manner that is both agent and time- dependent, and can generalize to unseen agent combinations. The attention heads in ball eevecc are somewhat analogous edge types, but, importantly, they do not require priori knowledge about the relationships between the players.
336
337
338
339
340
341
342
Additionally, unlike recent works that use variational methods to train their generative models 1 21. we translate the qulti-agent trajectory modeling problem into a classification task, which allows us to train model by strictly maximizing the likelihood of the data. As result, we do not make any assumptions about the distributions of the trajectories nor do we need to set any priors over latent variables. Zheng al. [10] also predicted binned trajectories, but they used recurrent convolutional neural network to predict the trajectory for a single player trajectory at a time at each time step.
344
346
348
349
5.2
Transformers for multi-agent spatiotemporal modeling
350
Giuliari et al. 31 used a Transformer to forecast the trajectories of individual pedestrians, i.e., the model does not consider interactions between individuals. Yu et al. 2 used separute temporal and spatial Transformers to forecast the trajectories of multiple, interacting pedestrians. Specifically, the temporal Transformer processes the coordinates of each pedestrian independently (i.e., it does not model interactions), while the spatial Transformer, which is inspired by Graph Attention Networks
351
353
8

---

8 processes the pedestrians independently at each time step. Sanford et al. [32 used Transformer 355
356 to classify on-the-ball events from sequences soccer games; however, only the coordinates of the K nearest players to the ball were included in the input (along with the ball's coordinates). Further, the order of the included players was based on their average distance from the ball for given temporal window, which can lead to specific players changing position in the input between temporal
357
358
359
360
windows. As far as we are aware, baller2vec is the first Transformer capable of processing all agents simultaneously across time without imposing an order the agents.
Limitations
362
385
While baller2vec does not have a mechanism for handling unseen players, number of different solutions exist depending on the data available. For example, similar to what was proposed in Alcom 20) a model could be trained to map a vector of (e.g.. NCAA) statistics and physical measurements
384
385
386
to baller2vec embeddings. Alternativlly, if tracking data is available for the other league, a single
387
baller2vec model could be jointly trained on all the data.
368 At least two different factors may explain why including player identity as input baller2vec only led to relatively small performance improvements. First, both player and ball trajectories are fairly generic-players tend to move into open space, defenders tend move towards their man or
369
370
371
the ball, point guards tend to pass to their teammates, and so on. Further, the location of a player on the court is often indicative of their position, and players playing the same position tend have similar skills and physical attributes. As result, we might expect baller2vec to be able to make
372
373
374
reasonable guesses about a player's/ball's s/ballss trajectory just given the location of the players and the ball
375
on the court.
376
Second, baller2vec may be able infer the identities of the players directly from the spatiotemporal data. Unlike (batter leterrpiccherrr 2vec 20). which was trained on several seasons of Major League Baseball data, baller2vec only had access to half of one season's 's worth of NBA data for training. As result, player identity may be entangled with season-specific factors (e.g., certain rosters coaches) that are actually exogenous the player's intrinsic qualities, i.e., baller2vec
377
378
379
380
381
may be overfitting to the season. To provide an example, the Golden State Warriors ran a very specific kind of offense in the 2015-2016 season-breaking the previous record for most three-pointers made in the regular season by 15.4% and many basketball fans could probably recognize them from bird's 's eye view (i.e., without access to any identifying information). Given additional seasons of data,
382
383
384
385
baller2vec would no longer be able to exploit the implicit identifying information contained in
386
static lineups and coaching strategies, so including player identity in the input would likely be more beneficial in that case.
387
Conclusion
388
389
In this paper, we introduced baller2vec, a generalization of the standard Transformer that can model sequential data consisting of multiple, unordered entities each time step. As an architecture that both is computationally efficient and has powerful representational capabilities, we believe baller2vec represents an exciting new direction for MASM. As discussed in Section 6 training baller2ver on more training data may allow the model to more accurately factor players away from season-specific patterns. With additional data, more contextual information about agents (e.g.. a player's age. injury history, or minutes played in the game) and the game (c.g.. the time left in the period or the score difference) could be included as input, which might allow baller2vec to learn an even more complete model of the game of basketball. Although we only experimented with static, fully connected graphs here, baller2vec can easily be applied to more complex inputs- for example, sequence of graphs with changing nodes and edges- by adapting the self- attentitnio mask
390
381
382
383
394
385
396
387
398
389
400
tensor as appropriate. Lastly, as generative model (see Alcorn and Nguyen (3 for a full derivation),
baller2vec could be used for counterfactual simulations (e.g., assessing the impact of different rosters), or combined with a controller to discover optimal play designs through reinforcement learning.
402

---

References
404
405
[1] Panna Felsen, Patrick Lucey, and Sujoy Ganguly. Where will they go? predicting fine-grained adversarial multi-agent motion using conditional variational autoencoders. In Proceedings of the European Conference on Computer Vision (ECCV), pages 732-747, 2018.
407
408
[2] Eric Zhan, Stephan Zheng, Yisong Yue, Long Sha, and Patrick Lucey. Generating multi-agent trajectories using programmatic weak supervision. In International Conference on Learning Representations, 2019. URL http:::///ppeeeeeeeew.eeee
409
hAcFQ
[3] Raymond A Yeh, Alexander G Schwing, Jonathan Huang, and Kevin Murphy, Diverse genera- tion for multi-agent sports games. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4610-4619, 2019.
412
413
414
[4] Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In Advances in Neural Information Processing Systems, 2015.
416
417
[5] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- fion Processing Systems, pages
418
2017.
[6] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. /4165, 2020.
421
425
[7]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference Learning Representations, 2021. URL
424
425
427
428
[8] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
429
430
[9] Cunjun Yu, Xiao Ma. Jiawei Ren, Haiyu Zhao, and Shuai Yi. Spatio-temporal graph transformer networks for pedestrian trajectory prediction. In Proceedings of the European Conference on Computer Vision (ECCV), August 2020.
432
434
[10] Stephan Zheng, Yisong Yue, and Jennifer Hobbs. Generating long-term trajectories using deep hierarchical networks. Advances in Neural Information Processing Systems, 29:1543-1551, 2016.
436
437
[1I] Kazuki Irie, Albert Zeyer, Ralf Schlüter, and Hermann Ney. Language modeling with deep transformers. In Proc. Interspeech 2019, pages 3905-3909, 2019. doi: 10.21437/mntesspeech. 2019-2225, http:/t::/dd doi org/ 10. 21137 htwp://terrrrpeeeee2e.e-- 2019- 2225
438
439
[12] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2015.
441
442
[13] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. In Advances in Neural Information Processing Systems, 2016
443
444 [14] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder- decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.31 1/.11////////441179 URL hhtps //www. www.wacwweb.co.. wwwpbbooo.o.aoogggo/oooooooo D14-10///D11111
446
448
450
[15] Baskethall- 2015-16 nba player stats: Totals, February 2021. URL. https 2ttp:///nAAAao161eeee11ee totals. html
10

---

452
[16] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, 2013.
453
[17] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv: 1301311301..81,,.200 2013.
456
457
[18] Quoc Le and Tomas Mikolov. Distributed representations of sentences and documents. In International conference machine learning, pages 1188-1196, PMLR, 2014.
459
[19] lya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, 2014.
480
461
[20] Michael A Alcor. (batter pitcher) 2vec: Statistic-free talent modeling with neural player embeddings. In MIT Sloan Sports Analyties Conference, 2018.
462
465
[21] Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: 2018
[22] Alex Graves. Generating sequences with recurrent neural networks.
arXiv preprint
486
arXiv: 0850, 2013.
487
[23] Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014.
468
489
[24] Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. In International Conference Learning Representations, 2015.
[25] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. International Conference on Learning Representations, 2015.
472
473
[26]
Kihwan Kim, Matthias Grundmann, Ariel Shamir, lain Matthews, Jessica Hodgins, and Irfan Essa. Motion fields to predict play evolution in dynamic sport scenes. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 840-847. IEEE, 2010.
475
478
[27]
Hoang M Le, Yisong Yue, Peter Carr, and Patrick Lucey. Coordinated multi-agent imitation learning. In International Conference on Machine Learning, volume 70, pages 1995-2003, 2017.
477
478
479
[28] Hoang M Le, Peter Carr, Yisong Yue, and Patrick Lucey. Data-driven ghosting using deep
480
imitation learning. In MIT Sloan Sports Analytics Conference, 2017.
481
[29] Mengshi Qi, Jie Qin, Yu Wu, and Yi Yang. Imitative non-autoregresssvv modeling for trajectory forecasting and imputation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12736-12745, 2020.
483
484
[30] Eric Zhan, Albert Tseng, Yisong Yue, Adith Swaminathan, and Matthew Hausknecht. Learning calibratable policies using programmatic style-consistency. In International Conference on Machine Learning, pages 1001-1 1011. PMLR, 2020.
485
486
487
[31] Francesco Giuliari, Irtiza Hasan, Marco Cristani, and Fabio Galasso. Transformer networks for trajectory forecasting, In International Conference on Pattern Recognition, 2020.
488
489
[32] Ryan Sanford, Siavash Gorji, Luiz G Hafemann, Bahareh Pourbabace, and Mehrsan Javan. Group activity detection from trajectory and video data in soccer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 898- 899, 2020.
491
[33] Michael A. Alcom and Anh Nguyen. baller2vec+:; A look-ahead multi-entity transformer for modeling coordinated agents. arXiv preprint arXiv:2104. 1980, 2021.
494

---

Checklist
495
496
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
498
contributions and scope? [Yes]
489
(b) Did you descrihe the limitations of your work? [Yes] See Section 6
500
(c) Did you discuss any potential negative societal impacts of your work? [No] Our work
501
does not introduce new ethical challenges.
502
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
504
2. If you are including theoretical results...
505
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
506
(b) Did you include complete proofs of all theoretical results? [Yes]
507
3. If you ran experiments..
508
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
510
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
511
512
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
513
514
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
515
516
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
517
(a) If your work uses existing assets, did you cite the creators? [Yes]
518
(b) Did you mention the license of the assets? [No] We link directly to the dataset.
(c) Did you include any assets either in the supplemental material or as a URL? [Yes]
520
(d) Did you discuss whether and how consent was obtained from people whose data you're
using/curating? [N/A]
521
522
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
523
524
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
526
527
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
528
$29
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
12


Reviews:
[{"review_id": "n0SQrfQ-iKf", "reviewer": "Reviewer_ETWf", "summary": "The manuscript presents a modification to the original transformer architecture that allows modeling (un-)ordered sequences of unordered sets. The TxT self-attention masks for sequences of T inputs in the original transformer are replaced by TxKxTxK mask tensors for a sequence of T input sets with K unordered entities each.\nThe architecture is used  in an autoregressive fashion for modeling the trajectories of basketball players and the ball. At each time-step the input consists of a set of players, represented by an identity embedding, raw x and y coordinates, and an optional context feature which indicates the direction of the player’s frontcourt. In task P, the set of target labels indicate the 2d motion corresponding to each input player. In Task B, the output labels indicate the 3d motion of the ball.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed architecture addresses the interesting task of modeling trajectories of basketball players and ball during NBA games. Please find detailed comments in the sections below.\n\nStrengths\n1. The baller2vec model outperforms a strong graph-based recurrent neural network architecture, even with only a quarter of the training time.\n1. I like the presentation of limitations, e.g. the discussion of the potential of overfitting to particular plays that can identify a team.\n1. The network does not require the introduction of order among the players while also not relying on a GNN approach.\n1. The authors present some good ablations and interesting observations, e.g. that the player embedding seems to cluster players with similar attributes (such as athleticism or high shooting ranges)\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations/Questions\n1. No variance is reported for the experiments. Does this mean you evaluated a single run for each experiment? Without any estimate of the variance between random seeds, it is hard to get a stable estimate of relative performances of different models. Addressing this would take several more week-long experiments, which is maybe more a general issue with transformer architectures and their high computational demand.\n1. “z” is used both for the z-axis in the ball position and the input sets.\n1. Transformers are widely used, but a small general introduction or a figure of the overall architecture (maybe in the appendix) would make sense.\n1. I think some details of the training setup are missing, e.g. what is the interval between validation steps. This is not a big problem though, since the code is made available.\n1. DId you try to model sequences longer than 4s?\n", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "kXlKyPoMqy", "reviewer": "Reviewer_7Vsi", "summary": "The authors introduce a transformer-based approach to model ball and player-trajectories in basketball.  Advantages of this approach include the ability to perform multi-agent trajectory prediction without requiring a strict ordering.  The approach itself, is not novel- it is a simple application of transformer architecture.  Additionally, this approach is only demonstrated on a single dataset in a specific setting.  Furthermore, model performance is only reported in terms of perplexity, not RMSE on predicted paths, and this prevents fair comparison to past approaches.", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work is largely an application of transformer architecture to a team-sports setting.  There is minimal originality or innovation on the algorithm side.\n\nThis paper is a resubmission of the work to ICML 2021.  The only significant change is the addition of the \"naive baselines\" in Section 3.3.  While additional baselines are helpful, issues remain.  First, this is not a \"naive baseline\"- this is using a GRNN similar to Yeh et al. with additional transformer features.  Because this is not a standard approach, it cannot be judged if this is a poor architecture decision (or trained poorly) or how it compares to the proposed method.  Comparing to Yeh et al directly would be a better approach.  Additionally, a true \"naive baseline\" would use something like constant velocity prediction or dictionary lookup.  \n\nIn addition to the baseline not being properly employed and tested, there is the remaining issue that simply adding this one section does not make this a sufficiently different or improved submission.  This paper was rejected at ICML, and there has not been sufficient improvement to warrant acceptance here.\n\nModel performance is given in terms of perplexity, which is fine, but prevents comparison to other approaches.   Previous work in this area including Yeh et al, Felsen et al, etc. all report RMSE on the final trajectory.  It is understood that this approach allows for multiple potential outcomes, but then calculate RMSE on the top-k paths.  The unwillingness of the authors to report such a metric suggests that this approach does not compare favorably to past approaches.  \n\nOverall the comparison to past approaches (both in terms of metrics provided as well as which methods are compared) is severely lacking and a major shortcoming of the work.  Without comparing to other approaches (dictionary lookup, Yeh et al, Felsen et al, Social LSTM, Social GAN, etc.), I don't believe this paper can be accepted.\n\nFigure 5 and the surrounding analysis is probably not well suited for a NeurIPS publication.  The embeddings shown there are not substantially different than those obtained by other methods.  Furthermore, there is no way to demonstrate they are \"better\".  Such analysis is better suited for a sports-focused venue/publication.\n\nSports is a useful setting for studying multi-agent motion.  However, given the assertion of this work is that this method is broadly useful for multi-agent trajectory forecasting.  That has not been demonstrated as only a single dataset has been used.  To demonstrate its usefulness, other multi-agent datasets should be explored.  \n\nThe claim that GNNs limit personalization (line 32) is false.  Player embeddings or other approaches are easily incorporated into such frameworks.\n\nIn general the figures are not as useful as they could be- it is difficult to comprehend what the output trajectories look like.  Figures 3 and 4 are not particularly useful or illuminating either.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors mention that this approach does not have a mechanism for handling unseen players.  This is a significant limitation as other past approaches have been able to address such an issue.  This is further an issue because unseen players is a common issue in sport-player tracking.  It is an even more serious issue in more unconstrained settings such as pedestrian tracking.  The authors mention possible ways to address this, but without exploring these and demonstrating this severe limitation can be overcome, the usefulness of this method is severely limited.\n\nSimilarly, limitations and impact were discussed only in the context of sports.  Presumably this method should extend to multi-agent modeling in a variety of settings, many of which have surveillance-related applications.  Limitations and impact in such other domains is noticeably absent. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "GSCWb_g32Eo", "reviewer": "Reviewer_d55H", "summary": "This paper proposes a multi-entity generalization of the Tranformer architecture to address the problem of multi-agent spatiotemporal modeling (MASM). They demonstrate the performance of their model on two basketball-related tasks, and showcase the performance of their approach in comparison to standard baselines. \n\n", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contributions of the paper are as follows: \n- they propose a generalization of the Transformer architecture to be suitable for multi-agent spatio-temporal modeling. To my knowledge, this is the first demonstratation of an approach which can simultaneously reaon about multiple agents in space-time. \n- demonstration of efficacy of model on basketball dataset on 2 tasks (ball trajectory modeling, player trajectory prediction), comparing to standard baselines \n\nMain concerns: \n- Trajectory prediction and MASM are well-studied domains and it is somewhat surprising to see the use of naive baselines for comparison. It would have been prudent to compare the performance of their model against the most applicable SOTA methods \n- Evaluation limited to a single dataset (basketball). I would be curious to see how this approach might perform on other datasets and related tasks (e.g. human trajectory prediction).\n\nSee for example: \n“Pedestrian Trajectory Prediction Using Context-Augmented Transformer Networks”. This paper demonstrated the prediction results by using the Transformer network with two different information not only position data. At a high level, the method is conceptually similar to their mult-entity formalism. \n\nMinor comment: Please justify why you'll have cast the prediction problem as a classification problem ? \n\nConclusion: \nWhile the overall idea is interesting, the novelty of approach is not immediately clear and the evaluation is insufficient in its current forms. Comparisons to SOTA methods, other datasets, and generalization to other multi-agent problems would help strengthen the merits of the work. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above. ", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "-C1pVxsAC0", "reviewer": "Reviewer_GDru", "summary": "In this work, a Transformer architecture is proposed which generalizes the typical Transformer model to be able to leverage information across entities and time (previously, models only operated with information from one or the other). It does this with a new block-lower-triangular self-attention mask matrix (actually a tensor, but reshaped to work with existing software frameworks). The method is experimentally verified on a dataset of NBA basketball games and the trajectories of players and the ball.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The work tackles an important problem which can be applied to many domains.\n\n- The authors present a novel multi-entity generalization of the typical Transformer via a joint entity-temporal attention scheme, improving the applicability of Transformer models to domains which have multiple entities interacting across time.\n\n- The paper is written clearly. In particular, Fig. 2, 3, 4 were helpful in understanding the method.\n\n- There are many ablation studies which aid in understanding the sources of performance for the model.\n\nWeaknesses:\n\n- Related Work:\n    - There is an earlier work related to graph-based basketball player trajectory forecasting that is worth adding to the related work discussion [1].\n\n- Experiments:\n    - Using a 95%/5% trainval-test split seems quite heavy on the training/validation side, a more common split might be 70/30 or 80/20 (the smaller the testing size, the less diversity of scenarios the model is tested on).\n    - The implemented model is _huge_. Was it necessary for the player and ball input features (which I understand to be a few numbers encoded to 20-dim vectors, Line 171) to be processed by 3-layer MLPs with hundreds of nodes per layer? Does the ~70x increase in parameters result in significantly better performance? It would be interesting to see a comparison to Felsen et al. [1] or other similar works even if they do have significantly less parameters (runtime is perhaps more of a consideration here rather than training time as in Table 2).\n    - While the authors did implement some relevant baselines, there are other transformer models for trajectory forecasting (with open-sourced code) that would serve as good (and arguably important) comparisons. For instance: https://github.com/FGiuliari/Trajectory-Transformer (while it does not model interactions, it is still a worthwhile comparison to show the power of modeling interactions) or some of the other cited methods in Section 5.2.\n    - The experiments are only conducted on NBA data. It is difficult to make the claim that baller2vec is an effective learning algorithm for the general problem of MASM without showing its performance on other MASM problems (e.g., more typical pedestrian or traffic motion, such as that found in the nuScenes dataset, for example).\n\n[1] B. Ivanovic, E. Schmerling, K. Leung, and M. Pavone, “Generative Modeling of Multimodal Multi-Human Behavior,” in IEEE/RSJ Int. Conf. on Intelligent Robots & Systems, Madrid, Spain, 2018.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "While the authors have included a limitations section, they have not written a broader impact section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers generally held the opinion that the experimental evaluation of this method was insufficient for acceptance.
Multiple reviewers cited the lack of comparison to other methods. Multiple reviewers brought up the issue of not reporting variances for model predictions and this issue was not adequately addressed. Reviewers also pointed to a lack of comparisons with other high performing methods and comparison of this method on other datasets as being outstanding issues. 

As a result of these issues only one of four reviewers gave a final recommendation of acceptance and the ACs final recommendation is to not accept this paper.

While one of the reviewers had previously reviewed an older version of this manuscript at ICML, they were careful in their review of this submission and their opinion was not a decisive factor in making the final decision for this paper.
Decision: Reject