Energy-based Potential Games for Joint Motion Forecasting and Control
Authors: Christopher Diehl
Tobias Klosek
Martin Krueger
Nils Murzyn
Timo Osterburg
Torsten Bertram
Venue: CoRL 2023 Poster (2023)
Keywords: Trajectory Prediction
Multi-Agent Interaction
Game-Theoretic Motion Planning
Energy-based Model
Optimal Control
Autonomous Vehicles

Abstract: This work uses game theory as a mathematical framework to address interaction modeling in multi-agent motion forecasting and control. Despite its interpretability, applying game theory to real-world robotics, like automated driving, faces challenges such as unknown game parameters. To tackle these, we establish a connection between differential games, optimal control, and energy-based models, demonstrating how existing approaches can be unified under our proposed Energy-based Potential Game formulation. Building upon this, we introduce a new end-to-end learning application that combines neural networks for game-parameter inference with a differentiable game-theoretic optimization layer, acting as an inductive bias. The analysis provides empirical evidence that the game-theoretic layer adds interpretability and improves the predictive performance of various neural network backbones using two simulations and two real-world driving datasets.


Reviews:
[{"review_id": "heT8g7omml", "reviewer": "Reviewer_xj4d", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes method for multi-agent motion forecasting based on a new neural network layer. This new layer solves a certain optimization problem (I think it's finding a certain Nash Equilibrium) while being differentiable. When this layer is plugged into prior motion forecasting systems, the resulting method achieves better _joint_ performance on the three measured benchmarks.", "summary_of_recommendation": "Overall, I think the ideas presented are intriguing, especially since they are backed by seem like strong empirical results. The lack of clarity in the paper makes it hard to fully appreciate the paper's contributions, especially on the theoretical front. So, while I'm currently voting to reject the paper, I would be convinced to revise my score if the paper were revised to improve the clarity.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Poor: Difficult to follow and understand.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. Does the proposed method have more learned parameters than the baseline?\n2. How would the proposed method compare to a baseline that uses an generic differentiable optimization layer (e.g., OptNet) instead of EPOL?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update 8/17/23: Increased score to \"weak accept\", as previously noted in the discussion with authors.\n\n--------------\n\n**Quality**: Empirically, it seems like the results are strong. It was great to see the visualizations of the predictions in the supplemental videos. I also appreciated that Table 2 shows both the positive and negative results: the proposed method does worse on the marginal metrics, but better on the joint metrics.\n\n**Originality**:  The proposed EPOL layer is novel to the best of my knowledge. The paper derives this layer by combining ideas from prior work (differentiable optimization, potential difference games).\n\n**Clarity**: I font the paper difficult to read, to the extent that I likely missed some of the nuances illustrated in Fig. 1 and Fig 2. A few suggestions\n* In the abstract, it was unclear what problem was being solved. I think part of the trouble is that the paper is simultaneously focusing on a broad idea (multi-agent coordination) and a very specific application area (autonomous driving). One option would be to make the paper entirely about autonomous driving, and mention at the end that similar theoretical ideas might be applied to other domains. Another option would be to keep the focus more theoretical, and only introduce the autonomous driving examples in the experiments section.\n* Revising the second paragraph of the introduction might help a lot with this. In particular, the intro sentence would explain that that are these two prior approaches. The next few sentences could explain the pros/cons of these approaches. And the last sentence could remain as-is (how to combine the strengths of these methods).\n* In the contributions paragraph, I was unclear what the \"types\" of the contributions where. I.e., are they are new equation? A new neural network layer? A new algorithm for motion forecasting? (This is clarified later in the paper)\n* By the end of the introduction, I was still unclear what problem was being solved by the proposed method. The end of the first paragraph suggests it's going to be something about interpretability, the second paragraph suggests it's going to be some combination of neural networks and game theory.\n* The paper mentions multi-modality in a number of places, but I don't think it was ever formally defined. I'm assuming that this refers to a probability distribution with multiple modes (as opposed to a model that takes as input images + lidar + ...), but it's unclear what random variable this multi-modal distribution is modeling.\n* The footnote on page 3 seems intriguing, but I didn't understand it.\n* I'm a bit confused about the input to the function $C$. Eq. 2 suggests that it takes 2 inputs, an observation and a control/action. L127 uses $C$ with just one argument, the control/action. Then L141 uses it with 2 observation inputs.\n* In the figures, I'd recommend making sure that the captions (1) explain what is in the figure (e.g., for Fig 3, explain that the colored rectangles refer to cars(?), and that the lines correspond to predictions); and (2) what the reader should learn from the figure.\n\n\n------------------\n**Minor comments**\n* Run a grammar checker on the paper (e.g., Grammarly, copy-paste into a google doc)\n* L38, \"explicit strategies ... implicit strategies\" -- I was confused by this.\n* L47 -- L60: These pargraphs are great! They provide a nice description of the prior work, along with the key similarities/differences from the proposed method.\n* Try to avoid starting sentences with numbered citations (L70) or equations (L121)\n* Table 1: I'd recommend moving the \"footnotes\" to the bottom of the table, making each one a new line, and using a smaller font (perhaps \\small or \\tiny)\n* Define notation before using it for the first time (e.g., L128 should appear before L125).\n* L137 -- L144: For this section especially, it'd be good to add more intuition.\n* \"scene-consistent\" -- Where is this defined?\n* L171 -- I think ${h, m}$ should be a sequence, not a set (order does matter).\n* L240 -- Add commas to the numbers (\"60,338\")\n* L253 \"unpublished model VIBES\" -- Where can readers go to learn more about VIBES?\n* Table 2 -- What does \"+SC\" refer to?\n* L308 \"Without loss of generality\" -- Why does assuming a fixed number of agents not restrict the generality of the method?", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "bWWq_lS5sq", "reviewer": "Reviewer_LVGe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes to warm-start trajectory optimization with learned initial strategies and learned cost weights to enable efficient multi-modal predictions in multi-agent navigation scenarios. The approach is deployed on several benchmark datasets and evaluated across several motion forecasting backbones as well as performance metrics to yield improvements over selected baselines.", "summary_of_recommendation": "The paper takes a promising approach of combining context-dependent learned initializations and cost weights with down-stream trajectory optimization to improve motion forecasting, and attempts to enhance interpretability of the decision making. While the statistical analysis of the results should be improved, I’m leaning towards accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- What is the statistical significance of the difference between the reported scores of the proposed method and the baselines? \n- How reasonable is it to assume symmetry of the coupling cost terms between agents? Particularly for the exiD dataset, wouldn’t one expect to observe differences between a small car trying to merge from an on-ramp into the lane currently occupied by a large truck?\n- The ablation on the optimization step count is interesting, though it’s a bit surprising that 2 optimization steps are sufficient – have you considered variations of the step size alpha?\n- Could you provide more examples of interpretable weight-decodings similar to Figures 3 & 7 on more complex interaction scenarios?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Strengths:**\n\nThe application of learning-based methods to enhance conventional optimization-based planners is a very promising direction. Predicting context-dependent weights for the energy function and providing these to the potential user as a form of interpretable decision-making is a nice touch particularly for the autonomous driving scenarios considered in this work. The paper is well-written and the list of abbreviations in the appendix is very helpful for ease of readability.\n\n**Weaknesses:**\n- There are many instances where performance scores of the best and second-best methods are very close. Have you tested for statistical significance of these differences (e.g. non-overlapping error bands)?\n- The figures showing exiD trajectories are difficult to interpret as the content appears cramped. The video is a nice addition, while it can be difficult to understand which parts to focus on for the exiD rollouts – highlighting particular aspects e.g. via red circles would be helpful.\n- The related works currently do not cover model-based RL approaches that can learn to predict multi-agent interactions for applications in driving (e.g. [1, 2]) and it would be interesting to briefly discuss relations to works from this area.\n\n[1] Schwarting, Wilko, et al. \"Deep latent competition: Learning to race using visual control policies in latent space.” CoRL, 2021.\n\n[2] Sessa, Pier Giuseppe, et al. \"Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computation.\" ICML, 2022.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "JweRukOQtU", "reviewer": "Reviewer_ZfDq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a method that connects learning and game theoretic optimization by training models to provide initializations for a downstream energy based optimizer. This method enables the system to leverage the best of both worlds: strong initializations from prior data to bootstrap optimization and interpretable/controllable optimization performance using domain knowledge. ", "summary_of_recommendation": "Overall, I recommend accept as the paper is well-written, describes a novel idea and validates its claims with experimental results.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Timing comparison against baselines (see main review)\n\nResults against alternative EPO applications discussed in Table 1", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper clearly articulates a novel method for motion-forecasting and control and demonstrates its performance on real-world datasets. There are some weaknesses in the evaluation that would be great to see addressed. \nStrengths:\n* paper is well written\n* results outperform the baselines on tasks of interest\n* ablations which clearly demonstrate the important of the learned initialization\n\nWeaknesses:\n* alternative EPO applications discussed in Table 1 are not compared against\n* lack of timing comparison against baselines - how much is learning impacting the runtime during inference? ", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a learning-based method for multi-agent motion forecasting that feastures a new optimization layer that combines potential theoretic initialization and energy-based optimization. 

The reviewers have found that the idea is interesting and the method is well-represented. The concern on the extensiveness of evaluation has been improved throughout the rebuttal phase.
Decision: Accept (Poster)