Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper introduces a differential simulator for self-driving in urban areas and uses it to build an imitation learning approach.
strengths_and_weaknesses: ### Strengths 
* The paper introduces a differentiable simulator for self-driving.
* The paper evaluates a method for imitation learning based on differentiation through the dynamics model of the simulator.
* The method is evaluated against several baselines.
* The authors are planning to share their open-source implementation for training and evaluation on the Lyft motion planning dataset.

### Weaknesses 
* The paper lacks algorithmic novelty. Planning through a model has already been explored in prior work, for example, in Universal Planning Networks (Srinivas etal., 2018).
* The set of the baseline is not adequate. I would encourage the authors to compare with Adversarial Imitation Learning methods as well.
summary_of_recommendation: I would recommend the paper for acceptance solely for the reason that the authors promise to share their code. Having a good codebase to build further research on can be just as important for advancing the topic as introducing novel algorithms. After a quick googling, I wasn't able to find another repository that performs imitation learning on the Lyft dataset.
issues: As think that the paper suffers from a limited comparison to the prior work. Also, the method for imitation learning introduced in the paper has limited novelty. However, I would strongly encourage the authors to continue their work since even benchmarking IL methods for self-driving and sharing their implementations is a valuable contribution on its own.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The present work proposes a novel differentiable data-driven scenario simulation technique that generates driving scenarios based on real-world data. This simulator is incorporated as a differentiable module in a policy gradient-based driving policy learning task. The authors evaluate the resulting policy learning system in comparison to several policy learning baselines on the Lyft Motion Prediction Dataset.
strengths_and_weaknesses: __What I learned__
- I learned how differentiable driving scenario generators can be integrated into a policy learning pipeline.

__Strengths__
- Data-driven simulation engine improves the data efficiency of the training process.
- Differentiability of the simulator allows its use as a neural network module in a more complex fully differentiable pipeline.

__Weaknesses__
- The positioning and motivation of the work is not fully clear, are you focussing more on the simulator or the policy learning aspect? On the policy learning side, what is the ultimate goal of the paper? Mainly basic research in embodied AI for mobility? Or actual deployment of learned driving policies for Self-Driving Vehicles (SDVs)?
- Depending on the answers to the former, different (emphasis of the) evaluations may be necessary. For example, focusing on improved navigation behavior of real-world SDVs might require comparisons with methods based on trajectory-based optimization or other AV planning techniques such as Multi-Policy Decision Making (Cunningham et al., 2015). Another evaluation-related issue is that it is done purely on a dataset rather than on any actual driving platform or at least in some video-game environment.
- Finally, the simulator itself is not compared with other scenario simulators such as TrafficSim.
- Some implementation details are missing, for example how was the data split (did you simply use the default split given in the dataset?).
-  The code is unfortunately not available at the time of submission. I still think it is great that the authors plan on releasing it.
- At times some smaller aspects are hard to understand (some examples are given as questions in the issues section). 
summary_of_recommendation: Overall, I see this work as going in a very interesting direction. My current objections are mainly motivation and evaluation related: The authors should also better clarify the motivation and positioning of the work. And the work needs to also evaluate the simulator in comparison to state-of-the-art. While the latter point may require significant work, I believe the motivation-related issues to be easily resolvable.
issues: __Main Paper__
- l. 79 / 80 What do you mean by "sim-to-real transfer" here? E.g. in the case of [23], the policy is not even retrained for the actual transfer.
- l. 129 & eq (2): In the current form of eq. (2), when the original and updated pose are the same, it seems like the new position becomes zero. Unless I am misunderstanding something, this does not look right.
- l. 138 do I understand correctly, that the simulator mainly allows for local perturbations of the expert trajectories rather then completely new scenarios in complex scenes such as intersections?
- l. 163: It is not clear to me why bias is introduced if the initial state of the trajectory is sampled from the expert? What characteristics of the expert driver's initial position are biased? Maybe something like speed?
- l. 198.: Could you provide implementation details on how exactly ChauffeurNet was modified (i.e. more details on your vectorization process for ChauffeurNet).

__Appendix__
- l. 16 How is the type embedding added? Simply via a one-hot representation?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a simulation-based, imitation learning-based approach for learning policies for urban driving scenarios from real-world human driving trajectories. To train the policies, a simulator is developed to replay traffic and simulate the ego vehicle, where the ego vehicle action is governed by a policy model utilizing PointNet and Multi-Head Attention layers. The policy model takes HD map, perceived objects' polygons, and the ego vehicle polygon as inputs and outputs a series of position coordinates and orientations as a trajectory. The model is trained by minimizing the error between the generated trajectories and those of experts. The results are compared with behavioral cloning and multi-step prediction methods.
strengths_and_weaknesses: The main strength of the paper is on presenting a differentiable traffic simulator from real-world driving data, which can potentially be used to generate synthesized driving scenarios and as a testbed for autonomous driving applications.  However, the paper does not provide sufficient details on the simulator and how the traffic is managed in the simulator. It seems that the traffic is simulated simply by replaying data. This could be problematic in many autonomous driving scenarios including scenarios, in which the ego vehicle behavior is slightly different from the expert demonstration.  For example, consider a driving case where the speed of the ego vehicle is marginally lower than the expert driver when stopping at a red traffic light. Without adjusting the traffic, this could result in a rear-end collision. This can become more critical when there is a significant difference between the policy and demonstration. For example, consider a case where the policy decides to make a valid lane change instead of remaining within the lane to follow the demonstration.  Given these cases, the application of the simulator seems to be restricted without considering a proper traffic management system. Additionally, the simulator's differentiablity seems to be narrowed down to the kinematic model of the ego vehicle, which is further reduced to a simple motion model. The simulator is not detailed adequately to judge whether perception inaccuracy or collision can lead to discontinuity or not. For example, does the simulator remains differentiable in case of collision or disappearance of an object? Lastly, it's not clear if the result is generalizable to different maps or scenarios.
summary_of_recommendation: The simulator is not detailed enough to judge its usability and impact on autonomous driving research and if there are any differences between the presented simulator and other simulators (e.g. [1]). The imitative policy seems to be just another model without being properly validated through ablation tests, generalizability results, and sufficient comparison results with SOTA. While less important, the provided comparison results also do not show a significant/clear improvement to BC. 
issues: 1. The comfort method is based on longitudinal acceleration. It's recommended to consider jerk and/or lateral acceleration instead as they are more representative of comfort in driving.
2. An off-road event threshold seems to be too large (4 m). Considering that lane width ranges between 3-4 m and vehicles usually drive in the center of lanes, the threshold could be reduced to 1.5-2 m to capture off-lane events. 
3. Referring to Table 1, BC and MS Prediction results seem to be very close to that of the proposed method when considering front and side collisions, but significantly higher for rear collisions. The reason behind this needs to be elaborated on in the paper. Should rear collisions be considered a failure?
4. The comparison results can be extended to more recent works including IRL/IM papers mentioned in the related work section.
5. There is no result to judge if the learned policy can generalize to different maps or scenarios. Further tests need to be provided to check the performance of the policy in a novel, but a similar map.
6. The simulator differentiably needs to be discussed in the case of collision, sudden appearance/disappearance of objects.
7. The related works need more recent methods to cover the whole problem including other end-to-end methods.
8. The difference between the presented simulator and the differentiable simulator presented in [1] needs to be highlighted.
9. Line 165-167, "To remove this bias we discard the first K timesteps ...", an experiment showing a comparison between with or without discarding K timesteps is helpful.
10. The introduction discusses covariant shifts between training and test distributions, but it is not clear how the proposed approach has addressed this problem.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper constructs a simulation environment based on real traffic data and train a traffic agent by means of imitation learning. To alleviate the covariate shift phenomenon, this paper introduces the “Differentiable simulator” method. The authors compare the performances of their method with some classical methods of imitation learning and show that their method has better performance in some indicators such as collision rate. In addition, the paper also carries out comparative tests on some reinforcement learning kinds of auxiliary rewards.
strengths_and_weaknesses: Strengths： 
The first is the investigated problem is very meaningful for autonomous driving, which is how to imitate a human-like traffic agent policy in a complex traffic scenario from the real dataset. 
The second is that this paper combines some current mainstream methods such as pointNet and differentiable forward loss for policy to improve the performance of the imitation learning.

The biggest weakness of the approach is, in my opinion, the lack of novelty. Indeed, all the core ideas of the contribution have been already proposed in existing methods. The idea of using differentiable simulator for reducing covariate shift was already proposed, with exactly the same loss style but different in the concrete form of calculation (this paper even use a more simplified form of Ls and La term, which in [35] are represent the backpropagation item towards policy parameter.) in [35] (in the paper). The use of reinforcement learning kinds of reward for helping revise the imitation learning policy behavior was already used in [1]. In addition, the baseline methods compared in this paper, One is behavior cloning inherently flawed in terms of covariate shift problem, which makes this comparison seem insufficiently persuasive. Therefore, the work appears to be very incremental at the moment, with only very small variations of existing methods.
[1] Bhattacharyya, R. , et al. "Modeling Human Driving Behavior through Generative Adversarial Imitation Learning." (2020).

summary_of_recommendation: Although the novelty is not enough, the investigated problem is very meaningful for autonomous driving. And the experimental result is convincing. 
issues: I think that this paper needs more empirical evidence that the proposed method outperforms other SOTA algorithm in imitation learning.
review_title: null

Meta-review: The authors provided thorough responses that addressed the reviewers' concerns. Following the authors' responses, all reviewers are generally positive. The AC endorses the reviewers' recommendation to accept the submission.
Decision: Accept (Poster)