baller2vec++: A Look-Ahead Multi-Entity Transformer For Modeling Coordinated Agents
Authors: Michael A. Alcorn
Anh Totti Nguyen
Venue: NeurIPS 2021 Submitted (2021)

Abstract: In many multi-agent spatiotemporal systems, the agents are under the influence of shared, unobserved variables (e.g., the play a team is executing in a game of basketball). As a result, the trajectories of the agents are often statistically dependent at any given time step; however, almost universally, multi-agent models implicitly assume the agents' trajectories are statistically independent at each time step. In this paper, we introduce baller2vec++, a multi-entity Transformer that can effectively model coordinated agents. Specifically, baller2vec++ applies a specially designed self-attention mask to a mixture of location and "look-ahead" trajectory sequences to learn the distributions of statistically dependent agent trajectories. We show that, unlike baller2vec (baller2vec++'s predecessor), baller2vec++ can learn to emulate the behavior of perfectly coordinated agents in a simulated toy dataset. Additionally, when modeling the trajectories of professional basketball players, baller2vec++ outperforms baller2vec by a wide margin.

baller2vectt; A Look-Ahead Multi-Entity Transformer For Modeling Coordinated Agents
Anonymous Author(s)
Affiliation Address
email
Abstract
In many multi-agent spatiotemporal systems, agents operate under the influence of shared, unobserved variables (e.g., the play a team is executing in a game of bas- ketball). result, the trajectories of the agents are often statistically dependent at any given time step: however, almost universally, multi-agent models implicitly assume the agents' trajectories are statistically independent at each time step. this paper, we introduce baller2vec+ multi-entity Transformer that can effec- tively model coordinated agents. Specifically, baller2vec++ applies a specially designed self-attention mask to mixture of location and "look-ahead" trajectory sequences to learn the distributions of statistically dependent agent trajectories. We show that, unlike baller2ver (baller hellee2vect''s predecessor), baller2vect* can learn to emulate the behavior of perfectly coordinated agents simulated toy dataset. Additionally, when modeling the trajectories of professional basketball players, baller2vect+ outperforms baller2vec by a wide margin.
5
11
13
Introduction and Related Work
15
Whether it is a team executing play in game of basketball, family navigating to an attraction in theme park, or friends posting about birthday party on social media platform, humans frequently coordinate their behavior in response to shared information. When this coordinating information is unobserved (which is often the case in many machine learning datasets), behaviors become correlated, i.e., the behavior of one individual at specific moment contains information about the behavior another individual the same time. In the context of modeling agent trajectories in multi-agent spatiotemporal systems, this property translates to the trajectories being statistically dependent at each time step. However, nearly all multi-agent spatiotemporal models (c.g., (T)6 implicitly (through their loss functions) assume the trajectories of the agents at each time step are statistically independent given the agents' previous locations (Figure T
16
17
18
individuals' observed
20
22
24
25 Zhan et al. 7 explicitly focused on modeling coordinated multi-agent trajectories, using "macro-
26 intents" " g that are shared across agents do so. The macro-intents are generated from separately trained recurrent neural network (RNN) that learns to predict future, coarse, "sta- tionary" location for each agent at each time step. The macro-intents for all the agents at specific time step are concatenated together to form single, shared, macro-intent variable, 30 which then provided as input to the trajectories-gcncrating model at that time step. However, similar to the previously mentioned multi-agent trajectory models, the macro-intents model im-
28
All data and code for the paper available at: <anonymized>.
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

32
plicitly assumes the macro-intents for the agents at each time step are statistically independent. Le., the macro-intent for one agent does not depend on the macro-intents of the other agents Further, the model still implicitly assumes the tra- jectories of the agents at each time step are independent, which is only true if the shared macro-intent vari- able perfectly captures all of the unob- served information that could cause the agents' trajectories to be corre- lated.
33
34
35
pl wdk.raal((aaralaaaaaa aadee Anra)
36
37
Art
BALD
39
41
F14.1
Zin2
31443
xiaa2
43
Notably, Social-BiGAT p does par- tially account for trajectory correla- tions through global adversarial loss. Specifically, the trajectories for each agent are separately passed through an encoder RNN, and the final hid- den states for each agent are then pro- cessed with graph attention network (GAT) [O]. The output of the GAT is then used an input to discrim- inator that classifies whether or not the input trajectories erated. This global adversarial loss, however, is only single component of the full Social-BiGAT loss func- tion, and other components of the loss function do implicitly make the inde- pendence assumption. Further, inter- estingly, adding the global discriminator to baseline model only improved the model's performance for one of six pedestrian datasets.
44
Figure 1: Left: most multi-agent systems implicitly sume the trajectories of the agents at each time step (A are conditionally independent given the agents' pre- vious locations (.)), Right: however, the various de- compositions of the joint probability of the trajectories, (note,
45
46
47
49
e.g.. p( A )p( Azr,2 A Art,3 A we omit the conditional 31:t,& terms for brevity), suggest more complex statistical dependencies between the agents" trajectories can exist (i.e., the independence assumption is an extremely strong one). Indeed, there are often shared un- observed variables influencing the spatiotemporal behaviors of agents as the play that the players basketball team are executing. or events occurring in a pedestrian envi- wohient-wwich suggests statistical dependencies between the agents' trajectories are likely.
51
59
real gen-
55
56
57
58
59
60
61
82
69
In this paper, we describe a novel multi-agent spatiotemporal model that integrates information about concurrent actions of agents to predict statistically dependent distributions of trajectories. Specifically, we extend the recently introduced multi-entity Transformer baller2vec [6] by: (I) augmenting its
85
86
input with a parallel sequence of "look-ahead" agent trajectories and (2) using a specially designed self-attention mask, which allows our model to exploit the chain rule of probability (Section 3). We
87
68
find that:
89
1. baller2vect+ is effective learning algorithm for modeling coordinated agents. Unlike baller2vec, baller2vect+ can learn to emulate perfectly coordinated agents from a simulated toy dataset (Section[5.] Further, baller2vec++ outperforms baller2vec by a wide margin (8.9%) when modeling the trajectories of professional basketball players (Section
70
71
72
73
2. baller2vec++ makes better predictions when conditioned concurrent trajectory infor- mation from other agents, supporting our proposition that the commonly used independence assumption for agent trajectories is overly strong (Section 5.2
75
76
3. Lastly, the joint probability assigned to sequence by baller2vec*+ is approximately permutation invariant with respect to the order the agents, i.c., baller2vect+ respects the properties of the chain rule (Section 5.33).
78
'See the authors'
implementation
here:
mul tittiatttt--a prograrrent arogrammatic- htt.g//ssss.a.aooooo...o.soaaart models/mace mac.ro vrnn. pp#11011

---

2
Background
80
81
2.1
Multi-agent trajectory modelling
82
Our problem description closely follows Alcorn and Nguyen 66. whom we quote here:
88
Let = = (1, B} be set indexing B agents and P = {pi, P2:....... pk C A be the K agents involved in particular sequence. {( Mr,1). (0.,2,9.2))...
[Let] C. (It.K, Vt.K) } [be] an unordered set of K coordinate pairs such that (Zi,kr (t,k) the coordinates for agent Pkه time step t. The ordered sequence of sets of coordinates = (C1, C2...., Cr), together with P. thus defines the trajectories for the K agents over T time steps.
85
87
88
In agget trajectory modeling, the goal is to model joint probability of the form:
P(AT1,1, Ari,,... AT, 11 21:6,2 T1a,K)
90 i... the joint probability of the K agents' trajectories A at time step given the agents' location 91 histories F1.tkk- We note here that the common practice of simultaneously predicting the trajectories 92 for all of the agents a specific time step is not required by theory. Using the chain rule probability,
93 the joint probability of the agents' trajectories can be factorized as, e.g.:
pl ( AZi.2 ATC.K
p( P(Ar,, AZ1, ATL AZL
94 where we omit the conditional historical trajectories for brevity, As result, it perfectly acceptable
95 generate trajectories agent-wise, using the previously generated trajectories as additional conditioning information when generating the trajectories for later agents (see Figure 2
98
AXL11
AFF.
F14,1
31133
Z
Figure 2: At inference time, model is not required to simultaneously generate the trajectories for all of the agents at a specific time step. An alternative strategy to allow the model to generate the agents' trajectories one a time, and let the model use the previously generated trajectories to inform the trajectories it generates for the remaining agents.
97
2.2 baller2vec (conditional) generative model.
baller2ves is recently described multi-entity Transformer that can model sequences of sets (the underlying data structure for multi-agent spatiotemporal systems), as opposed to sequences of individual inputs (like words in sentence). When used to model the game of basketball, the input at each time step for baller2vec is set of feature vectors where each feature vector contains information about the identity and location of player the court. baller2vec maps each input feature vector to an output feature vector, which is then used to "classify" the binned trajectory for that specific player at that specific time step.
99
101
103
105
Here, we provide probabilistic interpretation baller2vec, which establishes the theoret- ical grounds for using the chain rule to generate trajectories agent-wise at each time step in baller2vect+. Without loss of generality, we only consider one-dimensional trajectories for single agent here. To briefly summarize, the outputs of the softmax function over the binned
107

---

109 trajectories in baller2vec can be interpreted as mixture proportions for mixture of uniform distri-
110 butions with predetermined bounds that partition the Euclidean trajectory space. Further, because It-1 + Az, baller2ves is in fact conditional generative model that assigns probability to sequence of trajectories given the initial position of the agent, i.e., p( Ar2,..., Arr(s)). Using the chain rule, we decompose the joint probability of the trajectories as:
112
113
p( 1 A AUT) = p( (pAAs/AAr)) .:...p((ATTi AzI, AT2 ArT-1)
114
to reflect their temporal structure (we omit the conditional initial position term for brevity). Therefore, new trajectories can be generated from baller2vec with the following procedure (see Figure 3
115
116
1. First, sample one of the n different mixture components using the mixture proportions output from the classifier f (i.e., baller2vec) conditioned on the agent's current position, i.c., i ~ Categorical T2, T....., TT) where [TT1, T2, nn. = f(x)).
118
119
2. Next, sample trajectory from the uniform distribution associated with the sampled compo- nent, i.e., Ar, ~ (((a;,b).
120
121
3. Finally, add the sampled trajectory to the agent's input position generate the agent's position at the start of the next time step, i... 2++1 = Azt.
122
Let [AEmin: ATmaz) be an interval Ammin or Ar 2 ATmaz has zero density (i.e., such trajectories are humanly impossible). Let {aa, ba)) be set of intervals that par- tition the interval [Ariin: ATmaz) into n bins.
the real line such that any trajectory Ar
124
125
126
Le., U lax, bs) [Armin, Amraar) and i #
An a An b.) Az
128
j
(a,, bz) n [az, b,) = 0. Recall that the probability density function (PDF) for uniform
129
130
distribution with bounds -00 < < b < 00 is:
7~ Categorical(#1, ay Fa)
Sg
p(Ax)= = {6- b-a for Ar E b) otherwise
0
131
Letting G = b the PDF for mixture of
132
uniforms with these bounds is thus:
Figure 3: baller2vec can be viewed as a condi- tional generative model that assigns probability to a sequence of trajectories given the initial po- sitions of the agents. Here, we show graphical model depiction of a baller2vec model that gen- erates sequence of one-dimensional trajectories for single agent. Given the initial position of the agent (the circle containing ). one of n different uniform distributions (the square containing i is sampled using the mixture proportions (π;) output by baller2vec (f). The agent's trajectory (the di- amond containing ArT) is then sampled from the selected uniform distribution, which has bounds -00 < 0; < bj < co. At the start of the next time step, the agent's position is x22 = x1 + Ari- Maximizing the likelihood of baller2vec as a classifier over the binned trajectories is thus equiv- alent to maximizing its likelihood when assuming the trajectories are generated from mixture of uniform distributions that partition the Euclidean trajectory space (see Section 2.2 for details).
n
p(Ax)= Σ Tull Tul(A oz, b()
TTC (1)
where P(Ax) is the density assigned to Az
134 by the mixture, n; is the mixture proportion for the mixture component indexed by i (i.e.,
136
0 S म, < 1 and Σ ग. = 1), and U(Ar; a, b,)
137
is the density assigned Ax by the uniform dis-
138
tribution with bounds - 0o < 04 < b, < co.
139
Because the bounds of the uniform distribu- tions partition A Armax), Equation { reduces to:
140
141
P(Az)= TyCy
142 where Ar E (b)) (because the other uni- form distributions will assign a density of zero to
143
Az). The likelihood for data D (with DI N)
145 is then:
N C(D) II j=1
N
p p(Az)) II
j=1

---

148 where Ty.e" is the mixture proportion assigned to the component with Ar, E [ar, by) and Cj,a is the 147 associated density. Taking the negative logarithm of the likelihood gives:
N - -In(C(D)) = > In( Σ Ein(er)))
N
Because the bounds are fixed, the second summation is a constant, and Equation becomes:
In(C(D))
Σ In(Tye)) +C
149
where C C= -
In(c,,r). Therefore, minimizing the loss of baller2vec as a classifier of binned trajectories is equivalent to minimizing the loss of the model when assuming the trajectories generated from mixture of uniform distributions specified in Equation (.).
150
Model Architecture
152
ballez2vec mask
beller2vec look ahead mask
(=1
meeree attention information
additional layers
1-11-2
(=T
1
Figure 4: A naive strategy for learning to predict statistically dependent agent trajectories is to adapt the baller2vec self-attention mask so that baller2vec can "look ahead" at future positions of agents whose trajectories are generated prior to the agent being processed in the current time step. However, this doing so necessitates "seeing the future". For example, after the model attends to the blue agent's 's position at time step when processing the yellow agent time step the yellow agent's resultant feature vector contains information about the blue agent's future position. As result, when the model attends to yellow agent while processing the blue agent at the next level, the model seeing the future.
self-attention mask cannot be used with multi-layer Transformers because
We motivate our baller2vect architecture by first highlighting issue that arises in baller2vec when trying to model agent trajectories using the chain rule. The baller2vec self-attention mask be adapted that baller2vec "looks ahead" at the future positions of agents whose trajectories are generated prior to the agent being processed in the current time step (Figure(. However, this look-ahead self-attention mask only be used with the final layer of the Transformer; otherwise, the model needs to see the future (Figure 4 As a result, baller2vec is severely limited in the conditional distribution functions it can learn.
154
156
158
160
bal 1 (Figure overcomes this limitation by: (1) augmenting the baller2vec input with two other sets of feature vectors and (2) using specially designed self-attention mask. The three sets of feature vectors in baller2vect+ take the following forms:
162
163
1. =t,x = (le(pe), Ft.ks Mr.kk he,k)
(current location information)
184
2. 4r.k = ga. ((e(p))) 1: 1141 he,k ATC.. Alt.a)
("1ook-ahead" information) (initial location information)
3. T 9- ((((((p)), 21 V... V1 k)
5

---

U.A Bin(Ar Apra)
2
9 9o(le((s), Fe M0 He
d4.4 (u(le(pe)) Reta be bade Amrr Agua))
7
#2.
42.5
22.3
1=1
(=2
Figure 5: To learn statistically dependent agent trajectories, baller2vect+ uses a specially designed self-attention mask to simultaneously process three different sets of features vectors a single Transformer. The three sets of feature vectors consist of location feature vectors like those found in baller2vec (z,,k), look-ahead trajectory feature vectors (u,,&), and starting location feature vectors (rk: not shown). As can be seen in these partial depictions of baller2vec++ and the baller2vec++ self-attention mask, this design allows the model to integrate information about concurrent agent trajectories through multiple Transformer layers without seeing the future.
166
where 9z. Ju. and 9r are multilayer perceptrons (MLPs), e is an agent embedding layer, and hy.: is a vector of optional contextual features for agent pk: time step 24, is the same location feature vector used in baller2vec and contains information about a specific agent's identity and the agent's 's location at time step u is a "look-ahead" trajectory feature vector that contains information about a specific agent's identity, the agent's location at the next time step I + 1, and the agent's trajectory time step I, i.e., (I++1.A 1, Ut+,1, 1,+ ). Lastly, Tı is starting location feature vector that contains information about a specific agent's s identity and the agent's 's location at time step t = 1.
168
170
172
173
The тı feature vectors are necessary so that baller2vect+ can "see" the initial locations of all the agents when processing the agents agent-wise in the first time step.
174
These three sets of feature vectors are combined to form (K 2TK) x F matrix Z such that the first K rows consist of the K тı feature vectors, and the remaining 2T rows consist of the TK 2 and TK 4 feature vectors interleaved with one another, i.e., each 2,, is followed by its corresponding U,,K in the matrix. This matrix is passed into the Transformer along with the specially designed self-attention mask, which encodes the following dependencies (see Figure 5):
176
178
179
180
1. When processing Tk, baller2vect+ is exclusively allowed to "look" at each Thk (i.e., baller2vect+ cannot look at any location or look-ahead feature vectors when processing Tk₂)-
182
183
2. When processing kt,,k baller2vec++ is allowed "look" at: (i) each The (ii) any 2t.kk where (a) f1 < t2 or (b) t = to and ką < k2, and (ii) any ur, As where (a) t < 12 or (b)
184
185
4 = 12 and k) < k2-
3. When processing U22,k2 baller2vec++ is allowed to "look" at: (i) each Tki (ii) any 2t where (a) f1 < t2 or (b) t = 12 and k < k2, and (iii) any u, where (a) t < t2 or (b)
187
t to and k-
k2-
189
Each processed 2 feature vector is then passed through linear layer that is followed by a softmax, which gives probability distribution over the trajectory bins for agent p& at time step Similar to baller2vec, the loss for each sample is:
191
L
1n(f(2)).2-
192 where f(Z) (Z)),.)-1(or_.) is the probability assigned to the trajectory bin Vgy (where U2 = 193 Bin(Ar,, A is an integer from one to n²) by f, i.e. Equation (3) is the negative likelikeld (NLL) of the data according to the model.

---

195
Because any ordering of chain rule decomposition of a joint probability produces the same value, e.g.
196
= /rbe 3:tp:::aaa..aaaappapaaaaaaaasaaa
p(Ar,1)) Anr,
like MЛ we shuffled the order of the agents in each training sequence encourage the model to learn joint probabilities of the agent trajectories that are approximately permutation invariant with
198
respect to the ordering of the agents.
4
Experiments
201
We tested baller2vect+ on two different datasets. To highlight the pathological behavior of models that assume agent trajectories are statistically independent at each time step, we trained scaled down versions of baller2vect+ and baller2vec on toy dataset consisting of simulated trajectories for two perfectly coordinated agents. Additionally, to demonstrate the efficacy of baller2vec++ in real world settings, we trained baller2vec++ and baller2vec on a dataset of trajectories for professional basketball players.
203
205
207
4.1
Toy dataset
208
Each training sample was initialized with the agents starting at (-1,0) and (1, 0) on grid in random order (i.e., the first agent could be placed to either the left or the right of the origin). At each time step, one of nine actions (corresponding to the 3x3 grid surrounding the agent) was sampled from uniform distribution, and each of the agents was translated along this trajectory. This process was repeated for 20 time steps (see Figure ((aa for a sample).
209
210
212
213
4.2
Basketball dataset
214
We used the same National Basketball Association (NBA) datase employed by Alcom and Nguyen whom paraphrase here:
6..
216
The NBA dataset consists of trajectories from 631 games from the 2015-2016 season, which were split into 569/30/32 training/validationnesst games, respectively. During training, each sequence was sampled using the following procedure: (1) randomly select training game, (2) randomly select a starting time from the game, (3) take the following four seconds of data and downsample it to 5 Hz from the original 25 Hz, and then (4) randomly (with a probability of 0.5) rotate the court 180 This sampling procedure gave access to on the order of ~-22 million different (albeit overlapping) training sequences. For both the validation and test sets, ~1,000 different, non-overlapping sequences were selected for evaluation by dividing each game into [ non-overlapping chunks (where N is the number games), and using the starting four seconds from each chunk as the evaluation sequence.
217
218
219
220
221
222
223
224
225
226
227
228
4.3
Model
Our baller2veecae and baller2vec models for the basketball dataset closely followed 66, and so largely resemble the original Transformer architecture [12]. Specifically, the Transformer settings were: d 512 (the dimension of the input and output of each Transformer layer), eight attention heads, di = 2048 (the dimension of the inner feedforward layers), six layers, no dropout, and no positional encoding. Each MIP (i.e., 9z. Iu. and gr) had 128, 256, and 512 nodes in its three layers, respectively, and a ReLU nonlinearity following each of the first two layers. The player embeddings had 20 dimensions, and h was a binary variable indicating the side of the frontcourt for player Pk (i.c., the direction of his team's 's hoop) at time step t. Lastly, the 11 ft x 11 ft 2D Euclidean trajectory space was binned into 121 1
230
232
233
234
235
236
237
ft squares.
We used the Adam optimizer 14] with an initial learning rate of 10 10-6 6, 31 = 0.9, 82 = 0.999, and 1 to update the model parameters, of which there were 19 million. The learning rate was
238
http::///gthaab..o///oooo/////// Plyee-ooeeott
7

---

240
reduced to 10-² after 20 epochs of the validation loss not improving. Models were implemented in Py Torchh and trained on single NVIDIA GTX 1080 Ti GPU for ~600 epochs (seven days) where each epoch consisted of 20,000 training samples, and the validation set was used for early stopping.
241
For the toy dataset, we used scaled down versions of the basketball models with dynadel 128, four attention heads, de = 512, and two layers in the Transformer. Additionally, each MLP had two layers with 64 and 128 nodes, respectively. The models were trained for 50 epochs of 500 samples per epoch 10.5 minutes) using a single learning rate of 10-5.
244
245
248
5
Results
248
5.1
baller2vect+ can effectively model coordinated agents in both simulated and real settings
For the toy dataset, the training loss for baller2vec converged to ~2.2 15-mm In( i... the model 250
251
was simply independently guessing the trajectories for both agents at every time step. In contrast, the training loss for baller2vect+ converged to ~1.1 R 1--In -ln(g) 2, which is the expected loss
252
253
for model that perfectly learns the deterministic relationship between the agents' trajectories
254
(because the prediction for the second agent will always contribute - In(1.0) 0 to the loss).
255
When generating trajecto- ries with baller2vec, the agents are completely unco- ordinated, with each agent
258
257
258
259
following an independent random walk around the
260
261
grid (Figure (b)). In con- trast, trajectories generated by baller2vec++ display the same coordinated agent behavior as the training data (Figure (()).
263
(a) Training sample.
(b) baller2vec.
(c) baller2vec++.
265
Figure 6: When trained on a dataset of perfectly coordinated agent tra- jectories (a), the trajectories generated by baller2vec are completely uncoordinated (b) while the trajectories generated by baller2vec++ are perfectly coordinated (c). Animated versions can be found in the code repository.
267
For the basketball dataset, achieved an average NLL of 0.472 on the test set, 8.9% better than the average NLL for baller2vec (0.518) (see Figure ST for trajectories generated by baller2vec++ and baller2vell
268
baller2vect
269
271
272
273
As was observed in 66. the trajectory bin distributions for baller2vec become much more certain after observing portion of the sequence (Figure 7), which suggests baller2vec may be inferring some of the shared hidden variables (e.g.. plays) influencing the players. If that hypothesis was true, the performance gap between baller2vech of the sequence (before any shared hidden variables can be inferred by baller2vec). Indeed, the average NLL for baller2vecH in the first time step of each test set sequence (1.567) is 16.1 % better than the average NLL for baller2vec (1.869), while the average NLL for baller2vec++ in the last time step of each test set sequence (0.420) is only 9.7% better than the average NLL for baller2vec (0.465) (see Figure 7).
275
and baller2vec should be largest at the beginning
277
279
280
281
282
5.2 baller2vect+ makes better predictions when conditioned on concurrent trajectory information from other agents
283
284
Implicit in much of our discussion has been the intuition that providing a model with additional (relevant) information will improve its performance. To empirically test this conjecture, we compared the performance of baller2vec++ when predicting the trajectory of a specific basketball player placed in the first position of the player order (i.e.. when k 1) vs. predicting the trajectory for that same player placed in the last position (i.e., when h = 10). Specifically, for each player in each test sequence we calculated the NLL of the player's trajectory in the first time Step"wi with the player the
285
286
287
288
289
Because, as previously discussed, the benefits of baller2vech were most pronounced in the first time
step.
8

---

16.19
baller2ver
10
Leerávecr
Strfffeed everage NLL
1.5
0.8
Average NLL
p=0.997
0.6
0.4
0.5
0.2
0.0
10 11 Time stop
13
14
15
16
18
19
0.4 0.6 0.8
1.0
Unshuffed average NLL
Figure 7: Left: when modeling the trajectories of professional basketball players, the performance gap between baller2vect+ and baller2vec is largest the beginning of the sequence, before shared unobserved variables can be inferred by baller2vec. Each bar indicates model's average NLL over the entire test set for that particular time step. For full sequences, baller2vec++ outperforms baller2vec by 8.9%. Right: the joint probability assigned to a sequence by baller2vect+ is approximately permutation invariant with respect to the order of the agents. For each point, its a value indicates baller2vectt's average NLL for test set sequence using the original order of the agents in the sequence, while its y value indicates baller2vec*** average NLL. for the same sequence with the order of the agents shuffled. The shuffled average NLLs are highly correlated with their corresponding unshuffled average NLLs.
first position of the player order. Next, moved the player the last position of the player order, and then randomly shuffled the remaining nine players 10 times, calculating the NLL for the player 292 in the last position each time. Finally, we calculated the average percent change in the last position
291
293
NLLs relative to their corresponding first position NLLs. On average, moving player from the first
294
the last position improved the NLL for the player's trajectory by 14.6%.
295
5.3
The joint probability assigned to a sequence by baller2vec++ is approximately permutation invariant with respect to the order of the agents
296
To determine whether not baller2vect+ respects the fact that any ordering of chain rule decomposition of a joint probability produces the same value, we measured how much the average NLL for each test sequence in the basketball dataset varied when the order of the agents changed. Specifically, for each test set sequence, we shuffled the order of the agents 10 times. Then, for each permuted sequence, we calculated the percent errof in the average NLL relative the original, unshuffled sequence. Across all test sequences, the average percent error was only 1.55% Further, as can be seen in Figure 7 the shuffled average NLLs are highly correlated with their corresponding unshuffled average NLLs (Pearson correlation coefficient 0.997), i.c., the joint probability assigned sequence by baller2vect+ approximately permutation invariant with respect the order of the agents.
298
300
301
302
303
304
305
308
6
Conclusion and Future Work
In this paper, we have shown how the commonly used independence assumption of many multi-agent spatiotemporal models can severely limit their ability to learn to emulate coordinated agents. By relaxing this independence assumption in baller2vec, baller2vect+ was able to more accurately model the trajectories of professional basketball players. Models for other multi-agent spatiotemporal environments, such as pedestrian traffic (see [15 for survey) and vehicle traffic (e.g., (1)19), may
309
310
311
313
also benefit from the look-ahead approach used by baller2vect+. Additionally, the interleaved input design of baller2vecHe could be useful when modeling other systems involving many entities
315
interacting through time, such as social media platforms (c.g., 22 21)). However, confronting the
quadratic complexity of the Transformer attention mechanism the number of entities grows large
317
in these datasets an open problem, but recent work in sparse Transformers (c.g., (22 22) shows
encouraging progress.
Seee: https://en.e www.pediiiiiiiiiiiiiicm.m htt.pp/ppw........ppoopooooooooa

---

References
319
320
[1] Panna Felsen, Patrick Lucey, and Sujoy Ganguly. Where will they go? predicting fine-grained adversarial multi-agent motion using conditional variational autoencoders. In Proceedings of the European Conference on Computer Vision (ECCV), pages 732-747, 2018.
321
322
323
[2] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2255-2264, 2018.
324
325
326
[3]
Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, Hamid Rezatofighi, and Silvio Savarese. Sophie: An attentive gan for predicting paths compliant to social and physical constraints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1349- 1358, 2019.
327
328
329
330
[4] Raymond A Yeh, Alexander G Schwing, Jonathan Huang, and Kevin Murphy. Diverse genera- tion for multi-agent sports games. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4610-4619, 2019.
331
335
[5] Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, and Shuai Yi. Spatio-temporal graph transformer networks for pedestrian trajectory prediction. In Proceedings of the European Conference on Computer Vision (ECCV), August 2020.
335
[6] Michael A. Alcorn and Anh Nguyen. baller2vec: A multi-entity transformer for multi-agent spatiotemporal modeling. arXiv preprint arXiv:2102..3329,, 2021.
337
[7] Eric Zhan, Stephan Zheng, Yisong Yue, Long Sha, and Patrick Lucey. Generating multi-agent trajectories using programmatic weak supervision. In International Conference on Learning Representations, 2019. URL
339
hAcFQ
341
[8] Stephan Zheng, Yisong Yue, and Jennifer Hobbs. Generating long-term trajectories using deep hierarchical networks. Advances in Neural Information Processing Systems, 29:1543-1551, 2016.
343
[9] Vincet Kosaraju, Amir Sadeghian, Roberto Martin-Martín, Ian Reid, Hamid Rezatofighi, and Silvio Savarese. Social-bigat: Multimodal trajectory forecasting using bicycle-ge and graph attention networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue Liché-Bue, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019, URL https::/prooceeding: weaeeiiinnnnnnnuuineeeeee cctpppp/e.c///pp///11///// pdf
345
347
348
349
file/ /tte://04111333333334777//666663
350
[10] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks. International Conference on Learning Representations, 2018.
351
352
wtt.://wwweeeeeoorrrr/rraaaaapaa JXMpkkZZ accepted as poster.
353
[1I] Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint 19X.0:1906.082237, 2019.
354
355
356
[12] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informa- tion Processing Systems, pages
357
358
2017.
359
[13] Michael A Alcom. (batter pitcherr) 2vec:
Statistic-free talent modeling with neural player embeddings. In MIT Sloan Sports Analyties Conference, 2018
360
[14] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2015.
362
[15] Andrey Rudenko, Luigi Palmieri, Michael Herman, Kris M Kitani, Dariu M Gavrila, and Kai O Arras. Human motion trajectory prediction: survey. The International Journal of Robotics Research, 39(8):895-933 2020.
364
10

---

386
[16] Nachiket Deo and Mohan M Trivedi. Convolutional social pooling for vehicle trajectory prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1476, 2018.
367
[17] Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, and James Hays. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
370
372
373
[18]
Tianyang Zhao, Yifei Xu, Mathew Monfort, Wongun Choi, Chris Baker, Yibiao Zhao, Yizhou Wang, and Ying Nian Wu. Multi-agent tensor fusion for contextual trajectory prediction. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2019.
375
378
377
[19] Rohan Chandra, Uttaran Bhattacharya, Aniket Bera, and Dinesh Manocha. Traphic: Trajectory prediction in dense and heterogeneous traffic using weighted interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
378
379
380
[20]
Srijan Kumar, Xikun Zhang, and Jure Leskovec. Predicting dynamic embedding trajectory in temporal interaction networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1269-1278, 2019.
382
383
[21] Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, and Michael Bronstein. Temporal graph networks for deep learning on dynamic graphs. In ICML 2020 Workshop on Graph Representation Learning, 2020.
385
386
[22] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904. 10509, 2019.
387
[23] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences.
388
380
Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 17283-17297, Curran Associates, Inc., 2020. UtL/https:/pproeeddiie ineuu neurips Paper. pdf
392
[24]
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05 2020.
395
396
[25] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2020. URL /ttps::/openrrviee
397
398
[26] Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidimensional transformers. arXiv preprint arXiv:19122 12100, 2019.
400
401
[27] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? arXiv preprint a:X0v:2002.05095, 2021.
11

---

Checklist
403
404
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
406
contributions and scope? [Yes]
407
(b) Did you descrihe the limitations of your work? [Yes] See Section 6
408
(c) Did you discuss any potential negative societal impacts of your work? [No] Our work
409
does not introduce new ethical challenges.
410
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
412
2. If you are including theoretical results...
413
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
414
(b) Did you include complete proofs of all theoretical results? [Yes]
415
3. If you ran experiments..
416
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
419
420
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
421
422
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
425
(a) If your work uses existing assets, did you cite the creators? [Yes]
426
(b) Did you mention the license of the assets? [No] We link directly to the dataset.
(c) Did you include any assets either in the supplemental material or as a URL? [Yes]
428
(d) Did you discuss whether and how consent was obtained from people whose data you're
using/curating? [N/A]
429
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
431
432
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
434
435
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
436
437
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
12


Reviews:
[{"review_id": "b0vCBC7ptEq", "reviewer": "Reviewer_K2zL", "summary": "This work presents an extension to the existing baller2vec transformer architecture. In particular, a new attention mask is used internally that is lower-triangular (offset by one block diagonal) rather than block-lower-triangular. This enables baller2vec++ to attend over entities at both previous and one-step future timesteps (in particular by attending over one-step predictions). The method was demonstrated in two experimental scenarios, an illustrative toy example and a dataset of NBA basketball trajectories.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Compared to baller2vec, this work presents a novel extension to the self-attention mask which factors in one-step predictions and allows for statistical dependency between agents (which is not seen in prior works). Further, while permutation-variance might be an issue, the authors show in Section 5.3 that such a concern does not pan out empirically, which is reassuring.\n\nWeaknesses:\n- There are a significant amount of missing baselines in the experimental work. It is understood that this work is an extension of another, however this paper should be completely standalone, even if that requires bringing forward experiments and comparisons from the predecessor work.\n    - The same criticism applies to the related work, which is missing many references and relies on the reader having an intimate knowledge of the predecessor paper.\n\n- The experimental suite is limited to a simple illustrative scenario and data from NBA basketball games. However, in order to make claims such as the Section 5.1 heading, one would need to evaluate baller2vec++ on more datasets covering a wider range of scenarios (NBA basketball play is arguably at one end of the human motion spectrum). This is especially important to verify if the novel self-attention mask is necessary for modeling every day behavior such as human driving or walking within crowds, or if it is only empirically useful in a highly-coordinated and dynamic sport such as basketball.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have not adequately addressed the limitations and potential negative societal impact of their work, these sections are not present in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "Rf7XyRATI3i", "reviewer": "Reviewer_yWcc", "summary": "The authors proposed a multi-entity Transformer that can effectively model coordinated agents called baller2vec++. The proposed method applies a specially designed self-attention mask to a mixture of location and “look-ahead” trajectory sequences to learn the distributions of statistically dependent agent trajectories. Unlike the conventional baller2vec, the proposed method can learn to emulate the behavior of coordinated agents in a simulated toy dataset. Moreover, the proposed method modeled the trajectories of professional basketball players more accurately than the conventional baller2vec.\n\nThe strength of this paper seems to be 1) the experimental results outperforming the conventional baller2vec as described above, 2) relaxing the independence assumption of agents in baller2vec and adding “look ahead” information to the transformer input, and 3) the joint probability assigned to a sequence was approximately permutation invariant with respect to the order of the agents by the post-hoc analysis.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "However, there are two major weaknesses of this paper: 1) comparison only with the conventional baller2vec. Since trajectory prediction is a common task in machine learning, a comparison with other approaches (e.g., RNN-based) is required. 2) methodologically minor change (as described above 2 of the strength) and small theoretical contribution. Although relaxing the independence assumption of agents is an interesting idea, the modification of the previous model seems to be incremental in total. The theoretical ground for the permutation invariant will increase the value of this paper, in addition to the post-hoc analysis. For these reasons, I consider this paper does not meet the criteria for acceptance.\n\nMinor comments:\n\n33 i.e., the macro-intent for one agent does not depend on the macro-intents of the other agents. Macro-intents in [Zhan et al. 2019] may be shared among agents. The trajectory prediction among agents was not shared as the authors mentioned.\n\nFigure 4 is confusing between baller2vec and baller2vec++. Did baller2vec use  “look ahead” information?\n\n250 for the toy dataset, mainly training loss was evaluated. Please specify the reason (i.e., why not test loss) and whether the loss is training/test loss in the following sentences. For example, “254 (because the prediction for the second agent will always contribute − ln(1.0) = 0 to the loss).” is unknown whether training/test loss.\n\nIn Figure 6, was it better to predict different directions in the left figure (proposed method) from the ground truth? Moreover, I did not understand why the prediction loss for the second agent would always contribute − ln(1.0) = 0 in this case.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations were addressed. Societal impact was not addressed but the reason was described.", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "5gS3VI6mZ4", "reviewer": "Reviewer_VCp8", "summary": "Summary: \nThe paper proposed baller2vec++, utilizes transformer architecture for modeling multi-agent trajectories with dependencies. Applying a specially designed self-attention mask, baller2vec++ outperforms baller2vec on professional basketball dataset.", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \n1. The paper is clearly written.\n\nWeakness:\n1. The paper seems missing a related work section. \n\n2. The experiments is not convincing. Despite a toy dataset, only a Basketball dataset experiment is provided. \n\n3. The only baseline is baller2vec in the experiments. \n\n4. It resembles baller2vec and the novelty seems to be questionable.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "09abnhQocc", "reviewer": "Reviewer_GRxC", "summary": "The authors construct a framework based on multi-entity Transformers to handle the group dynamics (i.e. dependence between agents at each time step.) They implement a “look ahead” strategy to improve performance and coordination of agents.  This approach produces results much better than baller2vec, the model's predecessor.  The results are demonstrated on a basketball dataset and a toy set with perfect coordination.  While the performance on these datasets is good, there are a number of concerns regarding the paper including an absence of comparison to other methods, dependence on a fixed/\"binnable\" output space, only approximate permutation invariance, and ability to handle variable or a high number of entities.  As a result, the impact of this approach appears very limited.  \n", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The approach in the present work produces improved results over a past model (baller2vec) on a basketball dataset.  However, the significance of the work is severely limited by the lack of comparison to other methods and demonstrated performance on other datasets especially where there is less coordination and the number of entities may be large, variable, or noisy.  As currently formulated, the paper is more of an application and minimal extension of the Transformer framework to a very domain-specific problem which comes with strong assumptions (fixed entities, small discretizable spatial extent) that enables the approach to work.  Therefore the contribution to the broader domain of multi-agent trajectory forecasting appears severely limited.  \n\nThe overall text is clear enough to read.  However, I would suggest to the authors to spend more of the body of the paper focused on conducting experiments and comparisons to other approaches and ablation studies (knocking out players, adding poor data) as opposed to derivations of dependent probabilities pertaining to a past method.  \n\nNo physical measure (i.e. in terms of meters) is provided.  Given that this is about predicting agent's motion, this is a critical oversite.   While the model is probabilistic in nature, a physical measure of performance still needs to be provided in addition to NLL.  If you're trying to model motion to handle whether two pedestrians (or a pedestrian and a car) are going to collide, physical distance matters in addition to NLL of a given path.  A top-k measure in RMSE is needed.\n\nThe model is only tested on a single “real world” dataset plus a toy dataset with perfect coordination, both with a fixed number of entities who are present in every frame.  While this is true in a sports setting (or at least basketball- not true in football where a player may be sent off), this is not true in the general multi-agent setting.  For the work to be more impactful, its performance on other common multi-agent datasets should be demonstrated.  Note, the issue is not that a basketball/sport dataset was used or the primary focus- sport is a tremendous domain for studying multi-agent motion- it is simply that additional analysis in other domains was not performed.  If the author's believe the only real application space for this method is within sport where these assumptions hold, but that this approach is so much better than all other methods for this particular domain, then they should reformulate the paper to express this (both in terms of text and in comparison to other methods).  \n\nThe fact that the model is only approximately permutation invariant is highly problematic.  Many approaches out there (specifically those based on GNNs) are explicitly permutation equivariant.  While being approximately invariant may be acceptable in a sports setting, it could lead to large errors in more “natural” settings (like pedestrian detection) where movements are less coordinated and the number of agents is large and variable.  \n\nThe work is largely based on the baller2vec framework, which is a preprint.  The authors spend significant time discussing the “probabilistic interpretation” of this model which is fairly obvious given that it is largely an implementation of the transformer architecture.  \n\nThe “binned trajectory” approach that is used here dramatically reduces the applicability and usefulness of the approach in any setting outside of the sports domain where the spatial domain is fixed.  \n\nThe authors discuss the issue with the quadratic dependence on the number of entities in a Transformer framework, but they should also speak to variable number of entities.  \n\nHow does the model handle bad/missing data?  If there is a gap (or error) in a track, can the model still perform?  \n\nThere are no comparisons to other SOTA approaches (other than baller2vec).  Comparison to these other methods is critical, especially provided that baller2vec is only a preprint.  These other methods offer advantages in terms of handling variable/missing players, speed, personalization, and other features common in multi-agent task.  Social Bi-GAT is mentioned in the intro, but not compared to.\n\nThe statement in 22-24 that these models treat the trajectories as independent is untrue.  Something like Social-LSTM naturally contains information about past states.  Felsen 2018 explicitly conditioned on past trajectories to predict future motion.  The statement that the macro-intents in Zhan et al. are independent is not quite true either.  Although the dependency is not explicitly modeled, because their ordering is based on the group-structure evolution, there is an implicit dependence of the past motions of different agents impacting the other agents (i.e. the role assigned to a given agent, which impacts the prediction of the macro-intent, is very strongly influenced by the past motions of other remaining agents).\n\nIn the opening the authors discuss the coordination of agents being driven by a shared unobserved variable, like a play in basketball.  If this is the case, it should be possible to extract and perform some kind of analysis on this meta-state.  If the unobserved variable is changed, but the incoming trajectories are largely fixed (which is common at the start of many different plays), how will the subsequent trajectories change?  Probing this through ablation studies seems very important given the authors' claims.  On the flipside, what happens in less coordinated settings (like pedestrian tracking) where there is no explicit coordination?  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors state their work does not introduce any social challenges, which seems reasonable.\n\nThe authors speak to some limitations, but this has not been adequately addressed nor explored via experiments.  See comments above around the discrete output space, and number/noise/variability of the number of agents.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All ratings were "reject". The key problems the reviewers found with this paper is that it appears to be very close to baller2vec, the experimental comparisons to other models are not broad enough, there are insufficient ablation experiments, and generally there's a sense that the contribution is not substantial enough to warrant publication.

The authors were, however, very thorough and attentive in their rebuttal and discussion, addressing various reviewers' concerns as best as they could. While I felt this weighed in the authors' favor, it wasn't enough to sway the reviewers or me, and I feel rejection is still the correct decision.

I encourage the authors to follow up on some of the more involved suggestions the reviewers made, test on a wider range of datasets to assess how general the method is, try to distinguish this work a little more from baller2vec, etc. If the results improve, I believe the paper will be more viable in the future.
Decision: Reject