Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes to warm-start trajectory optimization with learned initial strategies and learned cost weights to enable efficient multi-modal predictions in multi-agent navigation scenarios. The approach is deployed on several benchmark datasets and evaluated across several motion forecasting backbones as well as performance metrics to yield improvements over selected baselines.
review: **Strengths:**

The application of learning-based methods to enhance conventional optimization-based planners is a very promising direction. Predicting context-dependent weights for the energy function and providing these to the potential user as a form of interpretable decision-making is a nice touch particularly for the autonomous driving scenarios considered in this work. The paper is well-written and the list of abbreviations in the appendix is very helpful for ease of readability.

**Weaknesses:**
- There are many instances where performance scores of the best and second-best methods are very close. Have you tested for statistical significance of these differences (e.g. non-overlapping error bands)?
- The figures showing exiD trajectories are difficult to interpret as the content appears cramped. The video is a nice addition, while it can be difficult to understand which parts to focus on for the exiD rollouts – highlighting particular aspects e.g. via red circles would be helpful.
- The related works currently do not cover model-based RL approaches that can learn to predict multi-agent interactions for applications in driving (e.g. [1, 2]) and it would be interesting to briefly discuss relations to works from this area.

[1] Schwarting, Wilko, et al. "Deep latent competition: Learning to race using visual control policies in latent space.” CoRL, 2021.

[2] Sessa, Pier Giuseppe, et al. "Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computation." ICML, 2022.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper takes a promising approach of combining context-dependent learned initializations and cost weights with down-stream trajectory optimization to improve motion forecasting, and attempts to enhance interpretability of the decision making. While the statistical analysis of the results should be improved, I’m leaning towards accept.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The authors propose a method that connects learning and game theoretic optimization by training models to provide initializations for a downstream energy based optimizer. This method enables the system to leverage the best of both worlds: strong initializations from prior data to bootstrap optimization and interpretable/controllable optimization performance using domain knowledge. 
review: This paper clearly articulates a novel method for motion-forecasting and control and demonstrates its performance on real-world datasets. There are some weaknesses in the evaluation that would be great to see addressed. 
Strengths:
* paper is well written
* results outperform the baselines on tasks of interest
* ablations which clearly demonstrate the important of the learned initialization

Weaknesses:
* alternative EPO applications discussed in Table 1 are not compared against
* lack of timing comparison against baselines - how much is learning impacting the runtime during inference? 
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, I recommend accept as the paper is well-written, describes a novel idea and validates its claims with experimental results.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes method for multi-agent motion forecasting based on a new neural network layer. This new layer solves a certain optimization problem (I think it's finding a certain Nash Equilibrium) while being differentiable. When this layer is plugged into prior motion forecasting systems, the resulting method achieves better _joint_ performance on the three measured benchmarks.
review: Update 8/17/23: Increased score to "weak accept", as previously noted in the discussion with authors.

--------------

**Quality**: Empirically, it seems like the results are strong. It was great to see the visualizations of the predictions in the supplemental videos. I also appreciated that Table 2 shows both the positive and negative results: the proposed method does worse on the marginal metrics, but better on the joint metrics.

**Originality**:  The proposed EPOL layer is novel to the best of my knowledge. The paper derives this layer by combining ideas from prior work (differentiable optimization, potential difference games).

**Clarity**: I font the paper difficult to read, to the extent that I likely missed some of the nuances illustrated in Fig. 1 and Fig 2. A few suggestions
* In the abstract, it was unclear what problem was being solved. I think part of the trouble is that the paper is simultaneously focusing on a broad idea (multi-agent coordination) and a very specific application area (autonomous driving). One option would be to make the paper entirely about autonomous driving, and mention at the end that similar theoretical ideas might be applied to other domains. Another option would be to keep the focus more theoretical, and only introduce the autonomous driving examples in the experiments section.
* Revising the second paragraph of the introduction might help a lot with this. In particular, the intro sentence would explain that that are these two prior approaches. The next few sentences could explain the pros/cons of these approaches. And the last sentence could remain as-is (how to combine the strengths of these methods).
* In the contributions paragraph, I was unclear what the "types" of the contributions where. I.e., are they are new equation? A new neural network layer? A new algorithm for motion forecasting? (This is clarified later in the paper)
* By the end of the introduction, I was still unclear what problem was being solved by the proposed method. The end of the first paragraph suggests it's going to be something about interpretability, the second paragraph suggests it's going to be some combination of neural networks and game theory.
* The paper mentions multi-modality in a number of places, but I don't think it was ever formally defined. I'm assuming that this refers to a probability distribution with multiple modes (as opposed to a model that takes as input images + lidar + ...), but it's unclear what random variable this multi-modal distribution is modeling.
* The footnote on page 3 seems intriguing, but I didn't understand it.
* I'm a bit confused about the input to the function $C$. Eq. 2 suggests that it takes 2 inputs, an observation and a control/action. L127 uses $C$ with just one argument, the control/action. Then L141 uses it with 2 observation inputs.
* In the figures, I'd recommend making sure that the captions (1) explain what is in the figure (e.g., for Fig 3, explain that the colored rectangles refer to cars(?), and that the lines correspond to predictions); and (2) what the reader should learn from the figure.


------------------
**Minor comments**
* Run a grammar checker on the paper (e.g., Grammarly, copy-paste into a google doc)
* L38, "explicit strategies ... implicit strategies" -- I was confused by this.
* L47 -- L60: These pargraphs are great! They provide a nice description of the prior work, along with the key similarities/differences from the proposed method.
* Try to avoid starting sentences with numbered citations (L70) or equations (L121)
* Table 1: I'd recommend moving the "footnotes" to the bottom of the table, making each one a new line, and using a smaller font (perhaps \small or \tiny)
* Define notation before using it for the first time (e.g., L128 should appear before L125).
* L137 -- L144: For this section especially, it'd be good to add more intuition.
* "scene-consistent" -- Where is this defined?
* L171 -- I think ${h, m}$ should be a sequence, not a set (order does matter).
* L240 -- Add commas to the numbers ("60,338")
* L253 "unpublished model VIBES" -- Where can readers go to learn more about VIBES?
* Table 2 -- What does "+SC" refer to?
* L308 "Without loss of generality" -- Why does assuming a fixed number of agents not restrict the generality of the method?
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, I think the ideas presented are intriguing, especially since they are backed by seem like strong empirical results. The lack of clarity in the paper makes it hard to fully appreciate the paper's contributions, especially on the theoretical front. So, while I'm currently voting to reject the paper, I would be convinced to revise my score if the paper were revised to improve the clarity.
review_title: null

Meta-review: This paper presents a learning-based method for multi-agent motion forecasting that feastures a new optimization layer that combines potential theoretic initialization and energy-based optimization. 

The reviewers have found that the idea is interesting and the method is well-represented. The concern on the extensiveness of evaluation has been improved throughout the rebuttal phase.
Decision: Accept (Poster)