Trajectory Prediction using Equivariant Continuous Convolution
Authors: Robin Walters
Jinxi Li
Rose Yu
Venue: ICLR 2021 Poster (2021)
Keywords: equivariant
symmetry
trajectory prediction
continuous convolution
argoverse

Abstract: Trajectory prediction is a critical part of many AI applications, for example, the safe operation of autonomous vehicles. However, current methods are prone to making inconsistent and physically unrealistic predictions. We leverage insights from  fluid dynamics to overcome this limitation by considering internal symmetry in real-world trajectories. We propose a novel model, Equivariant Continous COnvolution (ECCO) for improved trajectory prediction.  ECCO uses rotationally-equivariant continuous convolutions to embed the symmetries of the system. On both vehicle and pedestrian trajectory datasets, ECCO attains competitive accuracy  with significantly fewer parameters. It is also more sample efficient, generalizing automatically from few data points in any orientation.  Lastly, ECCO improves generalization with equivariance, resulting in more physically consistent predictions.   Our method provides a fresh perspective towards increasing trust and transparency in deep learning models. Our code and data can be found at https://github.com/Rose-STL-Lab/ECCO.

Published as a conference paper at ICLR 2021
TRAJECTORY PREDICTION USING EQUIVARIANT CON- TINUOUS CONVOLUTION
Robin Walters
Jinxi Li
Northeastern University
Northeastern University
r. wa wwwalters@norrhhsseeeeeee edu
11 j jinxi @@notteasteer.ccc edd:
Rose Yu
University of California, San Diego roseya@ucsd.
ABSTRACT
Trajectory prediction critical part of many Al applications, for example, the safe operation of autonomous vehicles. However, current methods are prone to making inconsistent and physically unrealistic predictions. We leverage insights from fluid dynamics to overcome this limitation by considering internal symme- try in real-world trajectories. We propose a novel model, Equivariant Continous COnvolution (ECCO) for improved trajectory prediction. ECCO uses rotationally- equivariant continuous convolutions to embed the symmetries of the system. On both vehicle and pedestrian trajectory datasets, ECCO attains competitive accu- racy with significantly fewer parameters. It is also more sample efficient, gen- eralizing automatically from few data points in any orientation. Lastly, ECCO improves generalization with equivariance, resulting in more physically consis- tent predictions. Our method provides fresh perspective towards increasing trust and transparency deep learning models. Our code and data can be found at https://githutt cwm Roose STL Lab/ECCO
INTRODUCTION
Trajectory prediction is one of the core tasks in AI, from the movement of basketball players to fluid particles to car traffic (Sanchez-Gonzale, et al. 2020: Gao et al.) 2020 Shah & Romijnders) 2016). common abstraction underlying these tasks the movement of many interacting agents, analogous to a many many-particle system. Therefore, understanding the states of these particles, their dynamics, and hidden interactions is critical to accurate and robust trajectory forecasting.
Even for purely physical systems such as in par- ticle physics, the complex interactions among large number of particles makes this dif- ficult problem. For vehicle or pedestrian tra- jectories, this challenge is further compounded with latent factors such as human psychology. Given these difficulties, current approaches re-
quire large amounts of training data and many model parameters. State-of-the-art methods in this domain such as Gao et al. (2020) are based on graph neural networks. They do not ex- ploit the physical properties of system and often make predictions which are not self-consistent or physically meaningful. Furthermore, they predict a single agent trajectory at a time instead of multiple agents simultaneously.
Figure 1: Car trajectories in two scenes. Though the entire scenes are not related by rotation, the circled areas are. ECCO exploits this symmetry to improve generalization and sample efficiency.
"Equal Contribution

---

Published as a conference paper at ICLR 2021
Our model is built upon a key insight of many-pan many--article systems pertaining to intricate internal sym- metry. Consider a model which predicts the trajectory of cars on a road. To be successful, such a model must understand the physical behavior of vehicles together with human psychology. It should distinguish left from right turns, and give consistent outputs for intersections rotated with different orientation. As shown in Figure I driver's velocity rotates with the entire scene, whereas vehicle interactions are invariant to such a rotation. Likewise, psychological factors such as reaction speed or attention may be considered vectors with prescribed transformation properties. Data augmenta- tion is a common practice to deal with rotational invariance, but it cannot guarantee invariance and requires longer training. Since rotation is a continuous group, augmentation requires sampling from infinitely many possible angles.
In this paper, we propose an equivariant continuous convolutional model, ECCO, for trajectory fore- casting. Continuous convolution generalizes discrete convolution and is adapted to data in many- particle systems with complex local interactions. [Ummenhofer et al. (2019) designed a model using continuous convolutions for particle-based fluid simulations. Meanwhile, equivariance to group symmetries has proven to be a powerful tool to integrate physical intuition in physical science appli- cations (Wang al. 2020 Brown & Lunter 2019 Kanwar et al. 2020). Here, test the hypothesis that an equivariant model can also capture internal symmetry in non-physical human behavior. Our model utilizes a novel weight sharing scheme, torus kernels, and is rotationally equivariant.
We evaluate our model on two real-world trajectory datasets: Argoverse autonomous vehicle dataset (Chang et al. 2019) and TrajNet++ pedestrian trajectory forecasting challenge (Kothari al. 2020) We demonstrate on par or better prediction accuracy to baseline models and data augmentation with fewer parameters, better sample efficiency, and stronger generalization properties. Lastly, we demonstrate theoretically and experimentally that our polar coordinate-indexed filters have lower equivariance discretization error due being better adapted the symmetry group.
Our main contributions are as follows:
We propose Equivariant Continous COnvolution (ECCO), a rotationally equivariant deep neural network that can capture internal symmetry in trajectories.
We design ECCO using a novel weight sharing scheme based on orbit decomposition and polar coordinate-indexed filters. We implement equivariance for both the standard and regular representation (SO(()))
On benchmark Argoverse and TrajNet++ datasets, ECCO demonstrates comparable accu- racy while enjoying better generalization, fewer parameters, and better sample complexity.
2 RELATED WORK
Trajectory Forecasting For vehicle trajectories, classic models in transportation include the Car- Following model (Pipes) 1966) and Intelligent Driver model (Kesting et al. 2010). Deep learning has also received considerable attention; for example, [iang et al. (2020) and Gao al. (2000) use graph neural networks to predict vehicle trajectories. Djuric et al. 2018 use rasterizations of the scene with CNN. See the review paper by Veress & Moussa (019 for deep learning in transportation. For human trajectory modeling, Alahi et al. (2016) propose Social LSTM to learn these human- human interactions. TrajNet (Sadeghian et al. 2018 and TrajNet++ (Kothari et al. (2020) introduce benchmarking for human trajectory forecasting. We refer readers to Rudenko et al. (2020) for a comprehensive survey. Nevertheless, many deep learning models are data-driven. They require large amounts of data, have many parameters, and can generate physically inconsistent predictions.
Continuous Convolution Continuous convolutions over point clouds (CtsConv) have been suc- cessfully applied to classification and segmentation tasks (Wang et al. 2018 Lei et al. [2119) Xu al. 2018: Wu et al., 2019 [Su et al., 2018 Li al. 2018: Hermosilla et al., 2018 Atzmon al.] Hua et al. 2018). More recently, few works have used continuous convolution for mod- cling trajectories or flows. For instance, Wang et al. (2018) uses CtsConv for inferring flow on LIDAR data. [Schenck & Fox (2018) and Ummenhofer et al. (2019) model fluid simulation using Ct.sonv. Closely related to our work is Ummenhoter et al. (2019), who design a continuous con- volution network for particle- based fluid simulations. However, they use ball-to-sphers mapping which is not well-adapted for rotational equivariance and only encode 3 frames input. Graph neural networks (GNNs) related strategy which have been used for modeling particle system
2018
2

---

Published as a conference paper at ICLR 2021
dynamics Gornhezz- et al. 2020). GNNs also permutation invariant, but they do not natively encode relative positions and local interaction as CtsConv-basee network does.
Equivariant and Invariant Deep Learning Developing neural nets that preserve symmetries has
been a fundamental task in image recognition (Cohen et al., 2019b, Weiler & Cesa 2019; Cohen
& Welling 2016al Chidester et al.] 2018 Lenc & Vedaldi 2015; Kondor & Trivedi 2018 Bao &
Song 2019 Worrall al.] 2017: Cohen & Welling 2016b Weiler al. 2018: Dieleman et al.
2016 Maron et al., 2020). Equivariant networks have also been used to predict dynamics: for
example, Wang et al. (2020) predicts fluid flow using Galilean equivariance but only for gridded data.
Fuchs et al. (2020) use SE(3) -equivaquivariant transformers to predict trajectories for small number of
particles regression task. As in this paper, both Bekkers (2020) and Finzi et al. (2020) address
the challenge of parameterizing kernel over continuous Lie groups Finzi et al. (2020) apply
their method to trajectory prediction on point clouds using a small number of points following strict physical laws. Worrall et al. (0177) also parameterizes convolutional kernels using polar coordinates, but maps these onto a rectilinear grid for application to image data. Weng al. (2018) address rotational equivariance by inferring a global canonicalization of the input. Similar to our work, Esteves et al. (2018) use functions evenly sampled on the circle, however, their features are only at single point whereas assign feature vectors to each point in a point cloud. TThoms et al. (2018) continuous convolutions. Unlike work, both Worrall et al. (2017) and Thomas al. (2018) define their kernels using harmonic functions. Our weight sharing method using orbits and stabilizers is simpler as does not require harmonic functions Clebsch-Gordon coefficients. Unlike previous work, we implement regular representation for the continuous rotation group SO(2) which is compatible with pointwise non- linearities and enjoys an empirical advantage over irreducible representations.
introduce Tensor Field Networks which are (0(3)-equivan)
BACKGROUND
We first review the necessary background of continuous convolution and rotational equivariance.
3.1 CONTINUOUS CONVOLUTION
Continuous convolution (CtsConv) generalizes the discrete convolution to point clouds. It provides an efficient and spatially aware way to model the interactions of nearby particles. Let p()) E g denote the feature vector of particle i. Thus f is a vector field which assigns to the points x(i) a vector in Ri The kernel of the convolution K: R²2 - Reat XCia is matrix field: for each point
x E RR2, K (x) is a Cout X Gin matrix. Let a be a radial local attention map with a(r) = -0 for r > R.
The output feature vector g(i) of particle i from the continous convolution is given by
g =CtsConvx,n(x,f f:x())
- -x I)x(() -x(0) Pb)
-
CtsConv is naturally equivariant to permutation of labels and is translation invariant. Equation | is closely related to graph neural network (GNN) (Kipf & Welling 2017 Battaglia et al. (2018). which is also permutation invariant. Here the graph is dynamic and implicit with nodes x and edges ei if ( - x| < R. Unlike GNN which applies the same weights to all neighbours, the kernel K depends on the relative position vector (i) x())
3.2 ROTATIONAL EQUIVARIANCE
Continuous convolution is not naturally rotationally equivariant. Fortunately, we can translate the technique of rotational equivariance CNNs to continuous convolutions. We use the language of Lie groups and their representations. For more background, see Hall (2015) and Knapp (2013).
More precisely, we denote the symmetry group of 2D rotations by SO(2) (Rots 0 2r). As a Lie group, it has both a group structure Rots, Rott, = Rot (e2))o2) which a continuous map with respect to the topological structure. As manifold, SO(2) is homomeomorphic to the circle 5 = (x E R² ||x 1). The group SO(2) act vector space R by specifying a representation map p: SO(2) + GL(R") which assigns to each element of SO(2) matrices (T.( The map p must be homomorphism
an element of the set of invertible
3

---

Published as a conference paper at ICLR 2021
((Rote) p(Roto, Roto2). For example, the standard representation P1 R is by 2 x 2 rotation matrices. The regular representation Preg on (2)))()) = {p S0(2) - R |s/l integrable) is Preg
= yp Rot---- Given input f with representation Pin and output with representation Pout. map F is (0)))))qqivarrian if
[Rotg)(s) (
F ([Pii(Rot) )f) Pout ((ott))(F).
Discrete CNNs are equivariant to group G' if the input, output, and hidden layers carry G-action and the linear layers and activation functions are all equivariant (Kondor & Trivedi 2018). One method for constructing equivariant discrete CNNs is steerable CNN (Cohen & Welling 2016b). Cohen et al. (2019a) derive general constraint for when a convolutional kernel K K"
X
G-equivariant. Assume G acts RR. and that IR Sout and R are G-representations Pout and Pin respectively, then K G-equivariant if for all E G, C.XERR,
K (gx) = Pout (g) (x)
For the group SO(2), Weiler & Cesa 2019) solve this constraint using circular harmonic functions give basis of discrete equivariant kernels. In contrast, our method is much simpler and uses orbits and stabilizers to create continuous convolution kernels.
ECCO: TRAJECTORY PREDICTION USING ROTATIONALLY EQUIVARIANT CONTINUOUS CONVOLUTIOON
In trajectory prediction, given historical position and velocity data of n. particles over 4n timesteps, we want predict their positions over the next tour timesteps. Denote the ground truth dynamics as E. which maps {( Vi-Gu:)) = Xg:++fou Motivated by the observation in Figure I we wish to learn model f that approximates the underlying dynamics while preserving the internal symmetry in the data, specifically rotational equivariance.
We introduce ECCO, model for trajectory prediction based on rotationally equivariant continuous convolution. We implement rotationally equivariant continuous convolutions using a weight sharing scheme based on orbit decomposition. We also describe equivariant per- per-particle linear layers which special of continuous convolution with radius R 0 analogous to 1xl convolutions in CNNs. Such layers are useful for passing information between layers from each particle to itself.
4.1 ECCO MODEL OVERVIEW
Ct.Conv
map
Conca
End
CtsConv
CtsConv
CtsConv
AP
Toun:
multi-ggen internetion
CtsConv
R
Enc
Jayens
belovior
Figure 2: Overview of model architecture. Past velocities are aggregated by an encoder Enc, To-
gether with map information this is then encoded by 3 CtsConvs Preg features. Then 1 1 CtsConvv layers are used to predict As. The predicted position êt+1 = A + where ă is numer- ically extrapolated using velocity and accleration. Since A is translation invariant, è equivariant.
The high-level architecture of ECCO illustrated in Figure 2 input, output, and hidden layers are all vector fields over the particles. Oftentimes, there is also
It is important to remember that the
4

---

Published as a conference paper at ICLR 2021
environmental information available in the form of road lane markers. Denote marker positions by Xmap and direction vectors by Vmap- This data thus also a particle field, but static.
To design equivariant network, must choose the group representation. This choice plays an important role in shaping the learned hidden states. We focus on two representations of SO(2); P1 and Preg- The representation P1 is that of our input features, and Prog for the hidden layers. For P1, For Preg. we further introduce a new operator, convolution
we constrain the kernel in Equation 1 with torus kernels.
In order to make continuous convolution rotationally equivariant, we translate the general condition for discrete CNNs developed in Weiler & Cesa (20T9) continuous convolution. We define the convolution kernel K in polar coordinates (68, r). Let R and R be SO(2) rreresessssntntoionnn Pout and Pin respectively, then the equivariance condition requires the kernel to satisfy
K(0 + o) = Pout Pout (Roty) ( P) (Rot,"
Imposing such constraint for continuous convolution requires us to develop an efficient weight sharing scheme for the kernels, which solve Equation
4.2 WEIGHT SHARING BY ORBITS AND STABILIZERS.
Given a point x E R and group G, the set Ox {gx g G} is the orbit of the point The set of orbits gives partition of I2 into the origin and circles of radius elements Gx {g gx x} fixing is called the stabilizer of the point x. We use the orbits and stabilizers to constrain the weights of K. Simply put, we share weights across orbits and constrain weights according to stabilizers, as shown in Figure Left.
The set of group
The ray [(0,r) r 2 0} is a fundamental domain for the action of G = S0(2) on base space R That is, D contains exactly one point from each orbit. We first define K (0,r) r) for each
r0,r) E D. Then we compute K K(0,r) r) from ((0,r) )) by setting e = 0 in Equation 3 as such
K(0, r) = Pout (Rotg) K (0, pm (Rot,')
(4)
For 0, the group acts freely (0, r), i.e. the stabilizer contains only the identity. This means that Equation imposes no additional constraints on (0, Thus K (0,,r) ) E Rew XCia is matrix of freely learnable weights.
For r = 0, however, the orbit ((0.0) is only one point. The stabilizer of (0,0) is all of G, which requires
K(0,0) 0) Pout (Rots) K(O, Pm (Rot,') for all 6. Thus K (00, 0) is an equivariant per-particle linear map Pin - Pout-
We can analytically solve
Equation 5 for (0,00) Table 1: Equivariant linear maps for K(0, 0). Trainable weights are E R and n: S - R, where S is the manifold underlying SO(2).
using representation the-
ory. Table | shows the
Pin
Pout P1
Pout = Preg
unique solutions for dif- ferent combinations of P1
P1
(a, + (ca, cb)
cos + cb sin
st f(0) cos cos(O)d) O0
subsection A.3
and Preg- For details see
Prog
P
Ssa I(0) sin (0)00)
SS x(0 ()
Note that 2D and 3D rota- tion equivariant continuous convolutions are implemented in Worrall al. (0117) and Thomas et al. 20188) respectively. They both use harmonic functions which require expensive evaluation of ana- lytic functions at each point. Instead, we provide a simpler solution. We require only knowledge of the orbits, stabilizers, and input/output representations. Additionally, we bypass Clebsch- Gordon decomposition used in Thomas et al, (2018) by mapping directly between the representations in our network. Next, describe an efficient implementation of equivariant continuous convolution.
4.3 POLAR COORDINATE KERNELS
Rotational equivariance informs our kernel discretization and implementation. We store the kernel K of continuous convolution 4-dimensional tensor by discretizing the domain. Specifically,
5

---

Published as a conference paper at ICLR 2021
K(0,r) r)
K (02)
((x)(02.01)
(or)dd
JoLESt
K(0.0)
Hoto K(O,7 r)
2m
X
Figure 3: Left: A torus kernel field K from a Preg-field to a Preg-field. The kernel is itself a field: at each point x in space the kernel (x) yields different matrix. We denote the (Φ2, Φ1) entry of the matrix at x = r0,r) by (0,, (02, Φ1). The matrices along the red sector are freely trainable. The matrices at all white sectors are determined by those in the red sector according to the circular shifting rule illustrated above. The matrix the red bullseye trainable but constrained to be circulant, i.c. preserved by the circular shifting rule. Right: The torus kernel acts on features which are functions on the circle. By cutting open the torus and features along the reg and orange lines we can identify the operation each point with matrix multiplication.
discretize IR² using polar coordinates with ky angular slices and k, radial steps. We then evaluate K at any r0, using bilinear interpolation from four closest polar grid points. This method accelerates computation since we do not need to use Equation 4 to repeatedly compute K(0, from K K(0, r). The special case of K(0, 0) results in polar grid with "bullseye" at the center (see Figure 3 Left).
We discretize angles finely and radii more coarsely. This choice is inspired by real-world observation than its exact distance, Our
that drivers tend to be more sensitive the angle of an incoming equivariant kernels are computationally efficient and have very few parameters. Moreover, we will discuss later in Section 4.5 that despite discretization, the use of polar coordinates allows for very low equivariance error.
4.4 HIDDEN LAYERS AS REGULAR REPRESENTATIONS
Regular representation Prey has shown better performance than Pt for finite groups (Cohen et al. 2019a Weiler & Cesal 2019). But the naive Preg {s: G + R} for an infinite group G too large to work with. We choose the space of square-integrable functions IG) It contains all irreducible representations of and is compatible with pointwise non-linearities.
Discretization. However, (S(()))) is still the manifold 5 underlying S0()) into kreg even intervals. We represent functions f E (2)S by the vector of values 2 )oosick.... We then evaluate f (Rota) using interpolation.
We resolve this by discretizing
We separate the number of angular slices ki and the size of the kernel hire- If we tie them together and set Ki krzgg this is equivalent to implementing cyclic group Ckieg symmetry with the regular representation. Then increasing hi would also increases kreg. which incurs more parameters.
Convolution with Torus Kernel. In addition to constraining the kernel K of Equation aas Pl. Preg poses an additional challenge as it is function on circle. We introduce a new operator from functions on the circle to functions on the circle called a torus kernel.
First, we replace input feature vectors in f E IR with elements of (SO)))))) The input feature f becomes Preg -ield, that is, for each x E IR? p(x) is a real-value function on the circle S1 R. For the kernel K. we replace the matrix field with a map K IR² + Preg @ Preg. Instead of a matrix, ((x) is a map 5 x S + R. Here (01, (2) E 5 x S plays the role of continuous matrix indices and we may consider K (x) (2) analogous to matrix entry. Topologically, St S is torus and hence call K K(x) torus kernel. The matrix multiplication K(x) ((x) in Equation

---

Published as a conference paper at ICLR 2021
must be replaced by the integral transform
((x) f(x (02) PIES!
K (02) ff)f(( ( (d1)d
which is linear transformation (²)SO(2)) (S) K(0., r) (02, Φ1) denotes the (#2, 01
entry of the matrix at point x = (f,r), see the illustration in Figure 3 Right. We compute Equation
for Prog Preg as K (Rots (02, Φ1) ((x)(02 U, Φ1 0). We use the same weight
sharing scheme as in Section 4.2
4.5 ANALYSIS: EQUIVARIANCE ERROR
The practical value of equivariant neural networks has been demonstrated in variety of domains. However, the- oretical analysis (Kondor & Trivedi 2018 Cohen al.
K22
20
10
2019a Maron et (222)) of continuous Lie group sym- metries is usually performed assuming continuous func- tions and using the integral representation of the convo- lution operator. In practice, discretization can cause the model to be not exactly equivariant, with some equiv- ariance error (EE) EE =
Equivartance Error (log
0.5
0.1
with respect to group transformations T and T' of input and output respectively (Wang al., 20200] A6). Rect- angular grids are well-suited to translations, but poorly- suited to rotations. The resulting equivariance error can be so large to practically undermine the advantages of theoretically equivariant network.
Re: number of angular slices flog scale)
Figure 4: Experimentally, we find kæ and expected equivariance error are in- versely proportional.
Our polar-coordinate indexed circular filters are designed specifically to adapt well to the rotational symmetry. In Figure 4 we demonstrate experimentally that expected EE is inversely proportional to the number of angular slices kg. For example, choosing ka 16 gives very low EE and does not increase the number of parameters. We also prove for Pi features that the equivariance error is low in expectation. See Appendix fo the precise statement and proof.
Proposition. Lei 22T/Ka, ka. and 0 be O rounded to nearest value in Zex, and 0 10 0|. Let F CtsConvK,R and = (Rotg). For some constant C, the expected EE bounded [x - (F)((((())))))))) T(x)) S sin(O)]C S 2n/C/k.
EXPERIMENTS
In this section, we present experiments in two different domains, traffic and pedestrian trajectory prediction, where interactions among agents are frequent and influential. We first introduce the statistics of the datasets and the evaluation metrics. Secondly, we compare different feature encoders and hidden feature representation types. Lastly, we compare our model with basclines.
5.1 EXPERIMENTAL SET UP
Dataset We discuss the performances of our models on (1) Argoverse autonomous vehicle motion forecasting (Chang et al. 2019). a recently released vehicle trajectory prediction benchmark, and (2) TrajNet++ pedestrian trajectory forecasting challenge (Kothari et al. 2020). For Argoverse, the task is to predict three-secont trajectories based on all vehicles history in the past seconds. We split 32K samples from the validation set our test set.
Baselines We compare against several state-of-the baselines used in Argoverse and TrajNet++ We use three original baselines from (Chang et al. 2019): Constant velocity, Nearest Neighbour, and Long Short Term Memory (LSTM). We also compare with a non-equivariant continuous con- volutional model, CtsConv (Ummenhofer al. 2019) and hierarchical GNN model VectorNet (Gao et al. 2020). Note that VectorNet only predicts a single agent at a time, which not directly comparable with ours. We include VectorNet reference nevertheless.

---

Published as a conference paper at ICLR 2021
Evaluation Metrics We use domain standard metrics to evaluate the trajectory prediction perfor- mance, including (1) Average Displacement Error (ADE): the average L2 displacement eπror for the whole 30 timestamps between prediction and ground truth, (2) Displacement Error sec- onds (DR@ts): the 1.2 displacement error at a given timestep t. DE@ts for the last timestamp is also called Final Displacement Error (FDE). For Argoverse, report ADE and DE@ts for / (1, 2, 3)- For TrajNet++, we report ADE and FDE.
5.2 PREDICTION PERFORMANCE COMPARISON
We evaluate the performance of different models from multiple aspects: forecasting accuracy, pa- rameter efficiency and the physical consistency of the predictions. The goal is to provide a compre- hensive view of various characteristics of our model guide practical deployment. See Appendix A.9 for an additional ablative study.
Forecasting Accuracy We compare the trajectory prediction accuracy across different models Argoverse and TrajNet+++ [Table 2 displays the prediction ADE and FDE comparision. We can see that ECCO with the regular representation Preg achieves on par or better forecasting accuracy on both datasets. Comparing ECCO and a non-equivariant counterpart of our model CtsConv, we observe a significant 14.8% improvement in forecasting accuracy. Compare with data augmentation, we also observe a 9% improvement over the non-equivariant CtsConv trained on augmented dataset. These results demonstrate the benefits of incorporating equivariance principles into deep learning models.
Argoverse
Tea++et++
#Param
Model
ADE
DE@ Is
DE@2s
DE@3s
ADE
FDE
Constant Velocity
3.86
2.43
5.10
7.91
1.39
2.86
Nearest Neighbor
3.49
2.02
4.98
7.84
1.38
2.79
LSTM
2.13
1.16
2.81
4.83
1.11
2.03
50.6K
CtsConv
1.85
0.99
2.42
4.32
0.86
1.79
1078. IK
CtsConv (Aug.)
1.77
0.96
2.31
4.05
1078. 1078.1K
P1-ECCO
1.70
0.93
2.22
3.89
0.88
1.83
51.4K
Prog-ECCO
1.62
0.89
2.12
3.68
0.84
1.76
129.8K
VectorNet
1.66
0.92
2.06
3.67
72K + Decoder
Table 2: Parameter efficiency and accuracy comparison. Number of parameters for each model and their detailed forecasting accuracy DE@ts. CtsConv(Aug.) is CtsConv trained with rotation augmented data.
Parameter Efficiency Another important feature in deploying deep learning models to embedded systems such as autonomous vehicles is parameter efficiency. We report the number of parameters in each of the models in Table 2] Compare with LSTM, our forecasting performance significantly better. CtsConv and VectorNet have competitive forecasting performance, but uses much more parameters than ECCO. By encoding equivariance into CtsConv, we drastically reduce the number of the parameters needed in our model. For VectorNet, Gao et al. (2020) only provided the number of parameters for their encoder; a fair decoder size can be estimated based on MLP using 59 polygraphs with each 64 dimensions input, predicting 30 timestamps. that is 113K.
Runtime and Memory Efficiency We compare the runtime and memory usage with VectorNet Gao et al. (2020). Since VectorNet is not open-sourced, we compare with version of VectorNet that we implement. Firstly, we compare floating point operations (FLOPs). VectorNet reported 0.041 GFL.Ps for the encoder part of their model alone, where n. is the number of predicted vehicles. We tested ECCO on a scene with 30 vehicles and approximately 180 lane marker nodes, which is similar to the test conditions used to compute FLOPs in Gao al. (2020). Our full model used 1.03 GFLOPs versus 1.23 GFLOPs for VectorNet's encoder. For runtimes on the same test machine, ECCO runs 684ms versus 1103ms for VectorNet. Another disadvantage of VectorNet is needing to reprocess the scene for each agent, whereas ECCO predicts all agents simultaneously. For memory usage the same test ECCO uses 296 MB and VectorNet uses 171 MB.
8

---

Published as a conference paper at ICLR 2021
Figure 6: The x.y-axes are the position (m). The dashed line represents the 2s past trajectory. The solid line represents the 3s prediction. Red represents the agent. Top row: The predictions are made on the original data. Bottom row: We rotate the whole scene by 160 and make predictions rotated data. From left to right are visualizations of ground truth, CtsConv, P1-ECCO, Preg ECCO.
Sample Efficiency A major benefit of incor- porating the inductive bias of equivariance is improve the sample efficiency of learning. For each sample which equivariant model trained on, it learns as were trained on all transformations of that sample by the sym- metry group (Wang et al. 2020)] Prop 3). Thus ECCO requires far fewer samples to learn from.
In Figure 5] we plot comparison of validation FDE over number of training samples and show the equivariant models converge faster.
Sampless
Figure 5: The learning curves on the validation Equivariant models converge faster using fewer samples than the non-equivariant models.
Physical Consistency We also visualize the predictions from ECCO and non-equivariant
CtsConv, as shown in Figurre 6 Top row visualizes the predictions on the original data. In the bottom row, rotate the whole scene by 160 and make predictions rotated data. This mimics the covariate shift in the real world. Note that CtsConv predicts inconsistently: a right turn in the top row but left turn after the scene has been rotated. We similar results for TrajNet++ (see Figure in Appendix(A.10) (A.
CONCLUSION
We propose Equivariant Continuous Convolution (ECCO), novel model for trajectory prediction by imposing symmetries as inductive biases. On two real-world vehicle and pedestrians trajectory datasets, ECCO attains competitive accuracy with significantly fewer parameters. It is also more sample efficient; generalizing automatically from few data points in any orientation. Lastly, equiv- ariance gives ECCO improved generalization performance. Our method provides fresh perspective towards increasing trust in deep learning models through guaranteed properties. Future directions include applying equivariance to probabilistic predictions with many possible trajectories, or devel- oping faster version of ECCO which does not require autoregressive computation. Moreover, our methods may be generalized from 2-dimensional space R". The orbit-stabilizer weight sharing scheme and discretized regular representation may be generalized by replacing SO(2) with SO(n), and polar coordinate kernels may be generalized using spherical coordinates.
ACKNOWLEDGEMENT
This work supported in part by Google Faculty Research Award, NSF Grant #2037745. and the U. S. Army Research Office under Grant W911NF-200 20-1-0334. Walters is supported by Postdoc- toral Fellowship from the Institute for Experiential AT at the Roux Institute.

---

Published as a conference paper at ICLR 2021
REFERENCES
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 961-971, 2016.
Matan Atzmon, Haggai Maron, and Yaron Lipman. Point convolutional neural networks by exten- sion operators. ACM Transactions on Graphics (TOG), 37(4):71, 2018.
Erkao Bao and Linqi Song. Equivariant neural networks and equivarification. arXiv preprint arXiv:1906.07172. 2019.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanzhez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv: 18011006600,, 2018.
Erik Bekkers. B-spline CNNs Lie groups. In International Conference on Learning Represen- tations, 2020.
Richard C Brown and Gerton Lunter. An equivariant Bayesian convolutional network predicts re- combination hotspots and accurately resolves binding motifs. Bioinformatics, 35(13):2177-2184, 2019.
Ming- Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hart- nett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, al. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 8748-8757, 2019.
Benjamin Chidester, Minh Do, and Jian Ma. Rotation equivariance and invariance in convolu- tional neural networks. arXiv preprint arXiv: 12301, 2018.
Taco S. Cohen and Max Welling. Group equivariant convolutional networks. In International
conference on machine learning (ICML), pp. 2990-2999, 2016a.
Taco Cohen and Max Welling. Steerable CNNs. arXiv preprint 16Xiv: 2016b.
Taco Cohen, Mario Geiger, and Maurice Weiler. general theory of equivariant CNNs on homo- geneous spaces. In Advances in Neural Information Processing Systems, pp. 9142-9153, 2019a.
Taco S. Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge equivariant convo- lutional networks and the icosahedral CNN. Proceedings of the 36th International Conference on Machine Learning (ICML), volume 97, pp. 1321-1330, 2019b.
Sander Dieleman, Jeffrey De Fauw, and Koray Kavukcuoglu. Exploiting eyclic symmetry in convo- lutional neural networks. In International Conference on Machine Learning (ICML), 2016.
Nemanja Djuric, Vladan Radosavljevic, Henggang Cui, Thi Nguyen, Fang-Chieh Chou, Tsung-Han Lin, and Jeff Schneider. Short-term motion prediction of traffic actors for autonomous driving using deep convolutional networks. arXiv preprint arXiv: 3ati::888.005550 008.005199, 2018
Carlos Esteves, Christine Allen-Blanchette, Xiaowei Zhou, and Kostas Daniilidis. Polar transformer networks. In International Conference on Learning Representations (ICLR), 2018
Mare Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolu- tional neural networks for equivariance to Lie groups on arbitrary continuous data. arXiv preprint arXiv:2002. 12880, 2020.
Fabian B Fuchs, Daniel E Worrall, Volker Fischer, and Max Welling. (eSt33)ttranffrmmrt 3D tnootrrnnslttion equivariant attention networks. arXiv preprint arXiv:2006. 10500, 2020.
Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. Vectomet: Encoding HD maps and agent dynamics from vectorized representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
11525-11533 11533, 2020.
10

---

Published as a conference paper at ICLR 2021
Brian Hall. Lie groups, Lie algebras, and representations: elementary introduction, volume 222. Springer, 2015.
Pedro Hermosilla, Tobias Ritschel, Pere-Pau Vázquez, Alvar Vinacua, and Timo Ropinski. Monte carlo convolution for learning on non-uniformly sampled point clouds. ACM Transactions on Graphics (TOG), 37(6):1-1,, 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Binh-Sor Son Hua, Minh- Khoi Tran, and Sii-Kit Yeung. Pointwise convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 984-993, 2018.
Gurtej Kanwar, Michael S Albergo, Denis Boyda, Kyle Cranmer, Daniel Hackett, Sébastien Racanière, Danilo Jimenez Rezende, and Phiala E Shanahan. Equivariant flow-based sampling for lattice gauge theory. arXiv preprint arXiv:2003.0641 2020.
Arne Kesting, Martin Treiber, and Dirk Helbing. Enhanced intelligent driver model to access the impact of driving strategies on traffic capacity. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 368(1993):458554665, 2010.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In International Conference on Learning Representations (ICLR), 2017.
Anthony W Knapp. Lie groups beyond an introduction, volume 140. Springer Science & Business Media, 2013.
Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks the action of compact groups. Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80, pp. 2747-2755 2018.
Parth Kothari, Sven Kreiss, and Alexandre Alahi. Human trajectory forecasting in crowds: A deep learning perspective. arXiv preprint arXiv:2007.03669, 2020.
Huan Lei, Naveed Akhtar, and Ajmal Mian. Octree guided CNN with spherical kernels for 3D point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 9631-9640, 2019.
Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equiv- ariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2015.
Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenn: Convolu- tion on x-transformed points. In Advances in neural information processing systems, pp. 820-830, 2018.
Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning lane graph representations for motion forecasting. arXiv preprint 2rXi::2007.13332 2020.
Haggai Maron, Or Litany, Gal Chechik, and Ethan Fetaya. On learning sets of symmetric elements. In Proceedings of the 37th International Conference on Machine Learning (ICML), 2020.
Louis Albert Pipes. Car following models and the fundamental diagram of road traffic. Transporta- tion Research/UK/, 1966.
Andrey Rudenko, Luigi Palmieri. Michael Herman, Kris M Kitani, Dariu M Gavrila, and Kai O Arras. Human motion trajectory prediction: A survey. The International Journal of Robotics Research, 3988)8955-935 2020.
Amir Sadeghian, Vineet Kosaraju, Agrim Gupta, Silvio Savarese, and Alexandre Alahi, Trajnet: Towards a benchmark for human trajectory prediction. arXiv preprint, 2018
11

---

Published as a conference paper at ICLR 2021
Alvaro Sanchez-Gonzalee Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Pe- ter W Battaglia. Learning to simulate complex physics with graph networks. arXiv preprint arXiv:2002.09405. 2020.
Connor Schenck and Dieter Fox. Spnets: Differentiable fluid dynamies for deep neural networks. In Conference on Robot Learning, pp. 317-335, 2018.
Rajiv Shah and Rob Romijnders. Applying deep learning to basketball trajectories. arXiv preprint arXiv: 03793, 2016.
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, and Jan Kautz. Splatnet: Sparse lattice networks for point cloud processing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2530-2539, 2018.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and cunslation-equivarian neural networks for 3d point clouds. arXiv preprint arXiv:1802.00 08219, 2018.
Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, and Vladlen Koltun. Lagrangian fluid simu- lation with continuous convolutions. In International Conference on Learning Representations, 2019.
Matthew Veres and Medhat Moussa. Deep learning for intelligent transportation systems: A survey of emerging trends. IEEE Transactions on Intelligent transportation systems, 2019.
Rui Wang, Robin Walters, and Rose Yu. Incorporating symmetry into deep dynamics models for improved generalization. arXiv preprint arXiv:2002.03061, 2020.
Shenlong Wang, Simon Suo, Wei-Chiu Ma, Andrei Pokrovsky, and Raquel Urtasun. Deep para- metric continuous convolutional neural networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2589-2597, 2018.
Maurice Weiler and Gabriele Cesa. General E(2))equivariam steerable CNNs. In Advances in Neural Information Processing Systems (NeurIPS), pp. 14334 14345, 2019.
Maurice Weiler, Fred Hamprecht, and Martin Storath. Learning steerable filters for rotation equivariant CNNs. Computer Vision and Pattern Recognition (CVPR), 2018.
Xinshuo Weng, Shangxuan Wu, Fares Beainy, and Kris M Kitani. Rotational rectification network: Enabling pedestrian detection for mobile vision. In 2018 IEEE Winter Conference on Applications of Computer Vision ( (WACV), pp. 1084-1092. IEEE, 2018.
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5028-5032 2017.
Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 9621-9630 2019.
Shi Xingjian, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. In Ad- vances neural information processing systems, pp. 802-810, 2015.
Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. Spiderenn: Deep learning on point sets with parameterized convolutional filters. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 87-102, 2018.
12

---

Published as a conference paper at ICLR 2021
A
APPENDIX
A.
CONTINUOUS CONVOLUTION INVOLVING Preg
This section is more detailed version of Section 4.4
Define the input f to be Preg-field, that is, distribution over R² valued in Preg. Define K IR² - Preg @ Prog. After identifying SO(2) with its underlying manifold S", we can identify K (x) as map 8 5t R and f(x); S I. Define the integral transform
K(x) @ (02) = J ((x)( (1))
For y R2, define the convolution g K fy
-fe 2ZEX
g(y)
K(x) f(x + y)dx.
The re-operation parameterizes linear maps Preg + Preg and is thus analogous to matrix multiplica- tion. If we chose restrict our choice of x(02, Φ1) R(d2 Φi) for some function R: S² then this becomes the circular convolution operation.
The (O(2)-action Preg by Rolo( l(o 0) induces action x: St St R by
Rota (x) Φ1) k(d2 0,01 -).
This, in turn, gives action the torus-field K by
Rota (=))(()( x1) K (R)) 0,01 -0.
Thus Equation 3 the convolutional kernel constraint, implies that K is equivariant if and only if
K (Roo)(()))(22 Φ (x) 0, di -).
We this define weight sharing scheme described in Section
3.2
The cases of continuous
convolution P1 + Preg and Preg + may be derived similarly.
A.2 COMPLEXITY OF CONVOLUTION WITH TORUS KERNEL
The complexity class of the convolution with torus kernel is O(n k?eg Cout Cin), where n is the number of particles, the regular representation is discretized into kieeg pieces, and the input and output contain Can and Cout copies of the regular representation respectively. We are not counting the complexity of the interpolation operation for looking up K ((.rr).
A.3 EQUIVARIANT PER-PARTICLE LINEAR LAYERS
Since this operation is pointwise, unlike positive radius continuous convolution, we cannot map between different irreducible representations of S0(2). Consider input Pin Pin-field I and output Pout-field O where Pin and Pout are finiteddimensional representations of SO(2). We define ( ( using the same W, an equivariant linear map, for each particle < < N. Denote the decomposition of Pin and Pout into irreducible representations of SO(2) as Pm 2 p @.... phr and Pout E p @ ... @ pf- respectively. By Schur's 's lemma, the equivariant linear map W Pin Pout defined by a block diagonal matrix with blocks (W& ) where Wx is an ix x jk matrix. That is, maps between different irreducible representations are zero and each map Pk Pk. is given by single scalar.
Per-particle linear mapping P1 + Preg and P1 + Preg. Since the input and output features are fieliss but the hidden features may be represented by Preg. need mappings between P1 and Preg. In all cases we pair continuous convolutions with dense per per-particle mappings, this we must describe per-particle mappings between (1 and Prog-
13

---

Published as a conference paper at ICLR 2021
By the Peter-Wey! theorem, (S)) = θ Pt. In the of SO(2), this decomposition is also called the Fourier decomposition or decomposition into circular harmonics. Most importantly, there is one copy of P1 inside of L (S)())). Hence, up to scalar, there is unique linear map i1 + (2)))) given by (a, b) + a cOs + boin
The reverse mapping pr (SO(() p1 is projection onto the pr summand and is given by the Fourier transform (f} = Uوg f(0) (0) Is f(0) si)(
Per-particle linear mapping Preg + Preg. Though Preg is not mnite-dimensional. the fact that decomposes into direct sum of irreducible representations means that we may take Pin = Pout = Preg above. Practically, however, it is easier to realize the linear equivariant map Pree - Pieg as convolution over S
0
k(0 x(0-))
deSI
where x(0) is an i x matrix of trainable weights, independent for each 0.
A.4 ENCODING INDIVIDUAL PARTICLE PAST BEHAVIOR
We can encode these individual attributes using a per vehicle LSTM (Hochreiter & Schmidhuber 1997.. Let x(()) denote the position of car time t. Denote fully connected LSTM cell by he: C. L h4-1, C--). Define hg = co = 0. We then use the concatenation of the hidden
states [h [ of all particles as Z R N R as the encoded per-vehicle latent features.
A.5 ENCODING PAST INTERACTIONS
In addition, we also encode past interactions of particles by introducing a continuous convolution LSTM. Similar to convLSTK TM we replace the fully connected layers of the original LSTM above with another operation Xingjian et al. (2015). While convISTM is well-suited for capturing spatially local interactions over time, requires gridded information. Since the particle system we consider are distributed in continuous space, we replace the standard convolution with rotation-equivariint continuous convolutions.
We can now define H,, C. = CtsConvLSTM(X, Hi-1, Ct-1) which is an LSTM cell using equiv- ariant continuous convolutions throughout. Note that in this case X,, Hi-1, Ct-1 are all particle feature fields, that is, functions (1...., n) R*
Define CtsConvI.TTM by
i oWWr x(1) + Wa h + Wio c 1 + bi)
ft = o (W,r fa *cts x + Wih * h-11 + Wfe 0 C- + bi)
G fi c + is tanh(W cr *ets x(9 + Wch * h1-1 + be)
0, oW... *cts x(() + Woh *ats h-1 + Woc + ba)
h = og tanh(c;),
where *ets denotes CtsConv. We then can use Hy, input feature for the prediction network.
A.6 EQUIVARIANCE ERROR
We prove the proposition in Section 4.5
Proposition. Let 2T/ kg. Let 0 be 0 rounded to nearest value Zo. Set 0 10 0). Assume particles samples uniformly in a ball of radius R with features f E pf. Let and K have entries sampled uniformly in [-a, a). Let the bullseye have radius Re R. Let F CtsConvk.R and T3 (Rott). Then the expected EE is bounded
Ex. x)) F(T(F) < [in( < 2
where C 4 (1 R2)Re)).
14

---

Published as a conference paper at ICLR 2021
Proof. We may compute for single particle x (w)r) and multiply result by n by linearity. We separate two cases: x in bullseye with probability R and x in angular slice with probability 1 R RR/ If x is in the bullseye, then there is equivariance error since K ((x) is scalar matrix.
Assume x is an angular sector.
For nearest interpolation, the equivariance emror is then
ll
Since (0) is length preserving, this is
|| (p1 K(x)f]
= -(0x((9)((x))((()))---
where 3 =0. We consider only single factor of P1 in f. The result will then be multiplied by c. Let
(
K11 K12
fi fz
((x)
f f=
12 h22
We can factor out an from (x) and an a from f and assume ki samples from Uniform([-1,1) One may then directly compute that Equation caciiionnequuu equuls
v ((h2) + k22) + (k1 - ks (2) (B)
This is bounded above by 4| si) 4| (f)). Collecting the above factors, this proves the bound C* (B)). The further bound follows by the first order bound,
sin(f)) S 10] S 2n /ko-
The relationship EE ~ 2TC/ko is visible in Figure 4
We can also see clearly the significance of the term [sin(0]] by plotting equivariance emoor against 0 as in Figure 7
Lquivariance trrar
Angle
Figure 7: The above plot is generated from random input and kernels. dependence of of EE on sin(
We can clearly see the
A.7 DATA DETAILS
Argoverse dataset includes 324K samples, which are split into 206K training data, 39K validation and 78K test set. All the samples real data extracted from Miami and Seattle, and the dataset provides HD maps of lanes in each city. Every sample contains data for seconds long, and is sampled in 10Hz frequency.
15

---

Published as a conference paper at ICLR 2021
TrajNet++ Real dataset contains 200K samples. All the tracking in this dataset is captured in both indoor and outdoor locations, for example, university, hotel, Zara, and train stations. Every sample in this dataset contains 21 timestamps, and the goal is to predict the 2D spatial positions for each pedestrain in the future 12 timestamps.
A.8 IMPLEMENTATION DETAILS
Argoverse dataset is not fully observed, so we only use cars with complete observation as our input. Since every sample doesn't it include the same number of cars, we only choose those scenes with less than or equal to 60 cars and insert dummy cars into them to achieve consistent car numbers. Tra- jNet++ Real dataset is also not fully observed. And here we keep our pedestrain number consistent to 160.
Moreover, for each car, we use the average velocity in the past 0.1 second as approximate to the current instant velocity, i.e. Vi (pr - pt-1)/2. As for map information, we only include center lanes with lane directions as features. Also, introduce dummy lane node into each scene to make lane numbers consistently equal to 650.
In TrajNet++ task, map information is included. And since pedestrians don't have speedometers to tell them exactly how fast they are moving as drivers, instead they depends more on the relative velocities and relative positions to other pedestrians, we tried different combination of features in ablative study besides only using history velocities.
Our models are all trained by Adam optimizer with base learning rate 0.001, and the gamma for linear rate scheduler is set to be 0.95. All our models without map information are trained for 15K iterations with batch size 16 and learning rate is updated every 300 iterations; for models with map information, we train them for 30K iterations with batch size 16 and learning rate is updated every 600 iterations.
For CtsConv, we set the layer sizes to be 32, 64, 64, 64, and kernel size 4 x 4 x 4; for Pı-ECCO, the layer sizes are 16, 32, 32, 32, hi is 16, ky is 3; for Preg-ECCO, we choose layer size 8, 16, 8, 8. ke 16, k, 3, and regular feature dimension is set to be 8. For Argoverse task, we set the CtsConv radius to be 40, and for TrajNet++ task we set be
A.9 ABLATIVE STUDY
We perform ablative study for ECCO to further diagnose different encoders, usage of HD maps and other model design choices.
Choice of encoders Unlike fluid simulations (Ummenhofer et al. 2019) where the dynamics are Markovian, human behavior exhibit long-term dependency. We experiment with three different coders refered to as Enc to model such long-term dependency: (1) concatenating the velocities from the past frames input feature, (2) passing the past velocities of each particle to the same LSTM to encode individual behavior of each particle, and (3) implementing continuous convolution LSTM encode past particle interactions. Our continuous convolution LSTM is similar ConvLSTM (Xingjian et al. 2015) but uses continuous convolutions instead of discrete gridded convolutions.
We use different encoders to time- agggeggeee features and compare their performances TTale 3p.
Use of HD Maps
In Table 4. we compare performance with and without map input features.
Choice of features for pedestrian Unlike vehicles, people do not have a velocity meter to tell him how fast they actually walk. We realize that people actually tend to adjust their velocities based on others' relative velocity and relative position. We experiment different combination of features Table 5). finding using relative velocities and relative positions as feature has the best performance.
A. QUALITATIVE RESULTS FOR TRAJNET++
Figure show qualitative results for TrajNet++. Note that the non-equivariant baseline (2nd column) depends highly on the global orientation whereas the ground truth and equivariant models do not.
16

---

Published as a conference paper at ICLR 2021
Argoverse
Tra+Net++
Encoder
ADE
DE@1s Is
DE@2s
DE@3s
ADE
FDE
Markovian
4.67
9.84
0.969
1.952
LSTM
2.05
1.06
2.51
4.71
0.909
1.909
CtsConvLSTN
3.98
2.02
5.11
8.40
0.962
1.941
DLSCMNVDLSTM
2.02
1.03
2.46
4.58
0,910
1.916
D-Concat(20t feats)
1.87
1.01
2.43
4.22
0.895
1.872
Table 3: Ablation study on encoders for Argoverse and TrajNet++. Markovian: Use the velocity from the most recent time step input feature. LSTM: Used LSTM to encode velocities of 20 timestamps. MisConvLSTMM Instead of dense layer, the gate functions in LSTM are replaced by CisConv. DLSC:: Replaced gate functions by CtsConv + Dense. D-Concat (20t feats):
Stacked velocities of 20 time steps as input.
w/o Map
w/ Map
Model
ADE
DE@ Is
DE@2 2s
DE@3s
ADE
DE@1s Is
DE@2s
DE@3s 3s
CtsConv
1.87
1.01
2.43
4.22
1.85
0.99
2.42
4.32
P1-FCCO
1.81
1.02
2.42
4.14
1.70
0.93
2.22
3.89
Preg Preg-ECCO
1.81
1.00
2.38
4.12
1.62
0.89
2.12
3.68
Table 4: Ablative study on HD maps for Argoverse. without HD Maps.
Prediction accuracy comparison with and
Velocity
Relative Position
Accelcration
ADE
FDE
Absolute
0.92
1.95
Absolute
X
0.90
1.87
Relative
0.89
1.86
Relative
0.86
1.79
Table Ablative study on features for Traj++. Acceleration means whether we used acceleration to make numerically extrapolated position.
Figure 8: The x,y-axes are the position (m). The dashed line represents the 2s past trajectory. The solid line represents the 3s prediction. Red represents the agent. Top row: The predictions are made on the original data. Bottom row: We rotate the whole scene by 160 and make predictions rotated data. From left to right are visualizations of ground truth, CtsConv, PCCCO Preg ECCO. ECCO
17


Reviews:
[{"review_id": "wW8PTyG956", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The polar-coordinate convolution has been proposed by multiple previous literatures, e.g. “Polar Transformer Networks” and “Rotational Rectification Network: Enabling Pedestrian Detection for Mobile Vision”. \n\nFor self-driving system, one should be able to get the global pose of your vehicle, then correct the global rotation of your scene based on the global pose. This should be a more practical solution to this issue. To me, this paper is overcomplicated as of a solution to the problem.\n\nThe theory part is well-written and easy to understand.\n\nExperiment is not strong enough though. I want to see how each of your variances add up in increased accuracy in ECCO network, e.g. experiments about stacking up “Weight sharing by orbits and stabilizers”, “polar coordinate kernels” and “hidden layers as regular representations”. Also, for the final model comparison (Table 2), I would like to see how ECCO compares to state-of-the-art other than VectorNet.", "extra_scores": {}, "extra_text": {}}, {"review_id": "fG5bvF0ngQ8", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposes a continuous convolution equivariant with respect to SO2 for trajectory prediction.\nRotational equivariance is achieved by a weight sharing scheme within kernels in polar coordinates.\nThe proposed scheme approximates rotational equivariance well and has similar or better performance on\nthe trajectory prediction tasks than models without equivariance while using fewer parameters.\n\n\n\nScore:\n\nI tend to accept for this paper. Rotational equivariance is an important problem that can hinder the broad application of ConvNets to sparse and continuous data where often graph networks are used. \nTrajectory prediction is a good application for rotational equivariance and the paper is written very general allowing the method to be transferred to other problems.\nAlthough some interesting comparisons with existing methods are missing, overall the experiments can show well the advantages of the proposed convolutions and are convincing.\n\n\n\nStrengths:\n\nCombining weight sharing with interpolation is a good idea to avoid unnecessary evaluations of the kernel at runtime.\n\nFigure 3 is very helpful but the font size for the labels should be optimized.\n\nThe experiments about physical consistency and the equivariance error answer important questions about the method.\n\n\n\nWeaknesses:\n\nThere are no direct comparisons to other equivariant convolutions, e.g. a comparison with Tensor field networks.\nAlthough the presented method is specialized for SO2, a comparison would be helpful to demonstrate the efficiency of the implementation. \n\nThere is also no information about the runtime and memory efficiency of the convolutions.\nTable 2 shows the number of parameters but the different models may have a different memory footprint during inference.\n\nThe Background section is a bit short. Adding some references to the literature on rotational equivariance in this section would be helpful.\n\n\n\nQuestions:\n\nA7 mentions that all models are trained for the same number of iterations.\nDid all models converge within 15k/30k iterations?\nIf not what is the performance of each model after convergence?\n\nAre the models without equivariant convolutions trained with data augmentations?\n\nWhat is the complexity class of the convolution with torus kernel?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "CHZ5kER4y10", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "--- Summary\n\nThis paper presents a novel Equivariant Continous COnvolution (ECCO) method for vehicle and pedestrian trajectory prediction. ECCO extends the previous continuous convolution method and makes it rotationally equivariant. To achieve that, they constrain the convolution kernel function K and make only the K(0, r) component freely trainable, and use it to derive the other components. They also propose a torus kernel that can be used on functions on circles. They evaluated their approach on two real-world trajectory prediction datasets, Argoverse and TrajNet++. They compared to basic baselines including constant velocity, nearest neighbor, and LSTM, as well as CtsConv (non-equivariant continuous convolution) and VectorNet. Results show that ECCO achieves significantly lower prediction errors than CtsConv and has fewer model weights. However, their prediction error is slightly higher than VectorNet at 2s and 3s.\n\n\n--- Strengths\n\n- The proposed ECCO trajectory prediction approach is novel in that the previous approaches are either not rotationally equivariant or only run on a single actor.\n\n- Making convolution operations rotationally equivariant can be very useful to many applications where rotations on the input aren't supposed to affect the outputs. The ECCO method proposed in this paper is very general, and I think it can be applied to the other applications as well.\n\n- With weight-sharing, the ECCO method greatly reduces the number of weights. The evaluation result also shows that the reduction of the trainable weights makes the model more data efficient.\n\n\n--- Major issues and suggestions\n\n- Actually, most of the state-of-the-art trajectory prediction approaches are already rotationally equivariant because they usually rotate their input scene to have the target actor that they are predicting always face up (e.g., in VectorNet). I think the only non-equivariant approaches are those one-shot approaches that predict all actors in the scene simultaneously. This work is interesting in that it is both one-shot and rotationally equivariant. However, from the evaluation results, it is not clear to me why predicting all actors in the scene simultaneously is a desired property. ECCO in fact has higher prediction errors than VectorNet. The inference speed can be a potential win, but the inference speed numbers are shown in the evaluation, unfortunately.\n\n- The evaluation baseline setup paragraph says \"VectorNet only predicts a single agent at a time, which is not directly comparable with ours\". I don't get why it is not directly comparable. In fact, VectorNet has lower prediction errors than ECCO at 2s and 3s. If the inference speed is an issue, the paper should show it with numbers.\n\n- The \"Parameter Efficiency\" paragraph says \"The number of parameters has strong implications in memory usage, battery consumption and compute speed.\" Why not just measure and show the memory usage and compute speed numbers?\n\n- The paper claims that trajectories predicted by the ECCO method are more physically realistic, but I don't see any results in the paper demonstrating this.\n\n\n--- Other comments\n\n- Section 3.2 is a little hard to understand. For example, what is S^1, what is GL, and what is L^2(SO(2))? I think providing more background or some references would be useful.\n\n\n--- Response to author's answers\n\n> The “target actor points up” method can be used to predict multiple agents by running repeatedly, these predictions are each independent, whereas in reality future trajectories for vehicles are mutually dependent. Two cars may have independently likely trajectories, but if they intersect, they are unlikely to occur together.\n\nYes, this is a commonly used argument used by works that predict all actors in the scene simultaneously. However, whether the claimed benefit is achieved by the proposed model is yet to be proved. The fact that the proposed model has higher prediction errors than the state-of-the-art single-actor prediction model makes me doubt whether this is true.\n\nThank you for updating the paper with the inference time numbers. These results are really useful.\n\n> For runtimes on the same test machine, ECCO runs 684ms versus 1103ms for VectorNet.\n\n1103ms is really high a latency for any real-time robotics system. For completeness, could you also include the number of actors that are predicted by VectorNet in 1103ms?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1hUvPC9ZeAK", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present a novel approach to trajectory prediction, where they use rotationally-invariant continuous convolutions. They present background on such convolutions and discuss a way to efficiently store and compute them,\nI found the paper not really well written, with a number of notation not being explained or skipped. I went over several relevant citations that the authors provided which helped, but some parts still remain unclear to me. I may not have understood all the math presented, which is the reason I lowered my confidence.\n\nMore comments:\n- The writing could be improved significantly. The authors introduce a lot of notation and concepts without really defining them well. E.g., K goes from K(.) to K(.,.) to K(.)(.,.), and similarly for rho_1 and rho_reg. I may be missing some background knowledge here, but I do feel that this should be much smoother and better explained nevertheless.\n- E.g., here they also introduce phi, theta, and other notation, but what that corresponds to in the trajectory prediction aspect is not explained or discussed. Or T in Section 4.5.\n- The authors mention physically-consistent predictions, but the method doesn't really contribute directly to that aspect of the prediction problem. Nor is it evaluated and discussed deeper. This should be removed or discussed more.\n- Same goes for \"transparency\", it is not really discussed deeper.\n- The authors don't explain well what are the actual model inputs that they use. This is glossed over, while usually works spend a lot of time here. They just say that they use histories and map info, but how exactly is unclear.\n- The authors also say that, unlike some competitors, they predict for all actors. This was also unclear how exactly. Or how are interactions taken into account. The explanations are very vague indeed.\n- Why are the scenes in Fig 6 not exactly the same, some actor locations are different? And especially for CtsConv where the actor is completely different it seems?\n- Curious how was the training done? Did you have to do some augmentation with scene rotations and imposing of the constraints?\nIn general, the discussion can be improved significantly. I marked my grade as 5 as I don't want to block the paper just because I didn't understand all the items, but I would be fine with mark 4 as well assuming the other reviewers have concerns along similar lines as myself.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes and studies a new SO(2)-equivariant convolution layer for vehicle and pedestrian trajectory prediction. The experiments are detailed and demonstrate the effectiveness of the approach in relation to non-equivariant models.
Decision: Accept (Poster)