Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors present a novel approach to trajectory prediction, where they use rotationally-invariant continuous convolutions. They present background on such convolutions and discuss a way to efficiently store and compute them,
I found the paper not really well written, with a number of notation not being explained or skipped. I went over several relevant citations that the authors provided which helped, but some parts still remain unclear to me. I may not have understood all the math presented, which is the reason I lowered my confidence.

More comments:
- The writing could be improved significantly. The authors introduce a lot of notation and concepts without really defining them well. E.g., K goes from K(.) to K(.,.) to K(.)(.,.), and similarly for rho_1 and rho_reg. I may be missing some background knowledge here, but I do feel that this should be much smoother and better explained nevertheless.
- E.g., here they also introduce phi, theta, and other notation, but what that corresponds to in the trajectory prediction aspect is not explained or discussed. Or T in Section 4.5.
- The authors mention physically-consistent predictions, but the method doesn't really contribute directly to that aspect of the prediction problem. Nor is it evaluated and discussed deeper. This should be removed or discussed more.
- Same goes for "transparency", it is not really discussed deeper.
- The authors don't explain well what are the actual model inputs that they use. This is glossed over, while usually works spend a lot of time here. They just say that they use histories and map info, but how exactly is unclear.
- The authors also say that, unlike some competitors, they predict for all actors. This was also unclear how exactly. Or how are interactions taken into account. The explanations are very vague indeed.
- Why are the scenes in Fig 6 not exactly the same, some actor locations are different? And especially for CtsConv where the actor is completely different it seems?
- Curious how was the training done? Did you have to do some augmentation with scene rotations and imposing of the constraints?
In general, the discussion can be improved significantly. I marked my grade as 5 as I don't want to block the paper just because I didn't understand all the items, but I would be fine with mark 4 as well assuming the other reviewers have concerns along similar lines as myself.
review_title: Writing should be improved, many details missing

final_score: rating: 7: Good paper, accept
review_content: Summary:

The paper proposes a continuous convolution equivariant with respect to SO2 for trajectory prediction.
Rotational equivariance is achieved by a weight sharing scheme within kernels in polar coordinates.
The proposed scheme approximates rotational equivariance well and has similar or better performance on
the trajectory prediction tasks than models without equivariance while using fewer parameters.



Score:

I tend to accept for this paper. Rotational equivariance is an important problem that can hinder the broad application of ConvNets to sparse and continuous data where often graph networks are used. 
Trajectory prediction is a good application for rotational equivariance and the paper is written very general allowing the method to be transferred to other problems.
Although some interesting comparisons with existing methods are missing, overall the experiments can show well the advantages of the proposed convolutions and are convincing.



Strengths:

Combining weight sharing with interpolation is a good idea to avoid unnecessary evaluations of the kernel at runtime.

Figure 3 is very helpful but the font size for the labels should be optimized.

The experiments about physical consistency and the equivariance error answer important questions about the method.



Weaknesses:

There are no direct comparisons to other equivariant convolutions, e.g. a comparison with Tensor field networks.
Although the presented method is specialized for SO2, a comparison would be helpful to demonstrate the efficiency of the implementation. 

There is also no information about the runtime and memory efficiency of the convolutions.
Table 2 shows the number of parameters but the different models may have a different memory footprint during inference.

The Background section is a bit short. Adding some references to the literature on rotational equivariance in this section would be helpful.



Questions:

A7 mentions that all models are trained for the same number of iterations.
Did all models converge within 15k/30k iterations?
If not what is the performance of each model after convergence?

Are the models without equivariant convolutions trained with data augmentations?

What is the complexity class of the convolution with torus kernel?

review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The polar-coordinate convolution has been proposed by multiple previous literatures, e.g. “Polar Transformer Networks” and “Rotational Rectification Network: Enabling Pedestrian Detection for Mobile Vision”. 

For self-driving system, one should be able to get the global pose of your vehicle, then correct the global rotation of your scene based on the global pose. This should be a more practical solution to this issue. To me, this paper is overcomplicated as of a solution to the problem.

The theory part is well-written and easy to understand.

Experiment is not strong enough though. I want to see how each of your variances add up in increased accuracy in ECCO network, e.g. experiments about stacking up “Weight sharing by orbits and stabilizers”, “polar coordinate kernels” and “hidden layers as regular representations”. Also, for the final model comparison (Table 2), I would like to see how ECCO compares to state-of-the-art other than VectorNet.
review_title: A Well-written paper, but the idea is not new.

final_score: rating: 6: Marginally above acceptance threshold
review_content: --- Summary

This paper presents a novel Equivariant Continous COnvolution (ECCO) method for vehicle and pedestrian trajectory prediction. ECCO extends the previous continuous convolution method and makes it rotationally equivariant. To achieve that, they constrain the convolution kernel function K and make only the K(0, r) component freely trainable, and use it to derive the other components. They also propose a torus kernel that can be used on functions on circles. They evaluated their approach on two real-world trajectory prediction datasets, Argoverse and TrajNet++. They compared to basic baselines including constant velocity, nearest neighbor, and LSTM, as well as CtsConv (non-equivariant continuous convolution) and VectorNet. Results show that ECCO achieves significantly lower prediction errors than CtsConv and has fewer model weights. However, their prediction error is slightly higher than VectorNet at 2s and 3s.


--- Strengths

- The proposed ECCO trajectory prediction approach is novel in that the previous approaches are either not rotationally equivariant or only run on a single actor.

- Making convolution operations rotationally equivariant can be very useful to many applications where rotations on the input aren't supposed to affect the outputs. The ECCO method proposed in this paper is very general, and I think it can be applied to the other applications as well.

- With weight-sharing, the ECCO method greatly reduces the number of weights. The evaluation result also shows that the reduction of the trainable weights makes the model more data efficient.


--- Major issues and suggestions

- Actually, most of the state-of-the-art trajectory prediction approaches are already rotationally equivariant because they usually rotate their input scene to have the target actor that they are predicting always face up (e.g., in VectorNet). I think the only non-equivariant approaches are those one-shot approaches that predict all actors in the scene simultaneously. This work is interesting in that it is both one-shot and rotationally equivariant. However, from the evaluation results, it is not clear to me why predicting all actors in the scene simultaneously is a desired property. ECCO in fact has higher prediction errors than VectorNet. The inference speed can be a potential win, but the inference speed numbers are shown in the evaluation, unfortunately.

- The evaluation baseline setup paragraph says "VectorNet only predicts a single agent at a time, which is not directly comparable with ours". I don't get why it is not directly comparable. In fact, VectorNet has lower prediction errors than ECCO at 2s and 3s. If the inference speed is an issue, the paper should show it with numbers.

- The "Parameter Efficiency" paragraph says "The number of parameters has strong implications in memory usage, battery consumption and compute speed." Why not just measure and show the memory usage and compute speed numbers?

- The paper claims that trajectories predicted by the ECCO method are more physically realistic, but I don't see any results in the paper demonstrating this.


--- Other comments

- Section 3.2 is a little hard to understand. For example, what is S^1, what is GL, and what is L^2(SO(2))? I think providing more background or some references would be useful.


--- Response to author's answers

> The “target actor points up” method can be used to predict multiple agents by running repeatedly, these predictions are each independent, whereas in reality future trajectories for vehicles are mutually dependent. Two cars may have independently likely trajectories, but if they intersect, they are unlikely to occur together.

Yes, this is a commonly used argument used by works that predict all actors in the scene simultaneously. However, whether the claimed benefit is achieved by the proposed model is yet to be proved. The fact that the proposed model has higher prediction errors than the state-of-the-art single-actor prediction model makes me doubt whether this is true.

Thank you for updating the paper with the inference time numbers. These results are really useful.

> For runtimes on the same test machine, ECCO runs 684ms versus 1103ms for VectorNet.

1103ms is really high a latency for any real-time robotics system. For completeness, could you also include the number of actors that are predicted by VectorNet in 1103ms?

review_title: Interesting work but with flaws

Meta-review: The paper proposes and studies a new SO(2)-equivariant convolution layer for vehicle and pedestrian trajectory prediction. The experiments are detailed and demonstrate the effectiveness of the approach in relation to non-equivariant models.
Decision: Accept (Poster)