Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Reject: I recommend rejecting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper proposes PlanT, a transformer-based end-to-end planner that uses object-level representations. The idea is very simple, the transformer takes vehicle bounding boxes as well as a discretized route as input tokens and outputs the planned future waypoints. Additionally, they predict the future of other vehicles in the scene as an auxiliary task that is shown to be helpful for better representation learning. The contributions are the method itself as well as a metric to measure the explainability of the planner.
strengths_and_weaknesses: ### Strengths
- Paper is written clearly
- The motivation for sparse representations outlined in the abstract is fair.
- Method is simple and easy to understand
- The proposed metric to evaluate the explainability of a planner, RFDS, is interesting. However, I believe the usefulness of the metric to drive progress in the field is questionable, as it only gets at the attention placed on other objects.

### Weaknesses
- **Main weakness:** the paper claims as a contribution that they demonstrate that "a simple object-level representation is sufficient to encode all the information relevant for planning in urban driving environments". This claim is false, and the reason is that the proposed method assumes perfect perception. While this limitation is recognized in the limitations section, its importance is well beyond what the paper assumes. The reason why other papers use BEV dense representations is to *capture uncertainty*. When not having perfect perception and dealing with challenges such as occlusion, uncertainty on the current and future states of objects grows very significantly. Since the proposed method accesses the internal state of the simulator to know the object locations, speed, etc., there is no uncertainty in the current states of objects, and very little on future ones since CARLA's actors are overly simplistic. Since there is no uncertainty about object existence, all the information can be captured at the object level and thus the proposed method work best. However, this is not a realistic robotic application. Even if the authors would do experiments with a real object detector, CARLA is not the most suitable environment for that as perception is much easier than in the real-world.
- The paper also claims to be 5.3x faster than previous method. First, that is not obvious from Table 1. Moreover, even if the claim were true with the current experiments, we come back to the fact that the method doesn't perform object detection. Running object detection would impact inference time in two ways. First, the time to run the object detector. Second, the number of actors would increase significantly to attain an acceptable recall, further increasing runtime.
- It is also claimed that route is all the map representation needed. Again, I believe this is true for their experiments, but an flawed argument in reality. First, CARLA's scenarios are very simple and the other actors quite reactive. This makes it easy for the SDV to never have to steer off the route to perform mitigation maneuvers like a safe stop. Second, we come back to assuming perfect perception. Since the method has a privileged understanding of the scene there is much less uncertainty, which translates into the SDV never having to steer off the route.
- While the attention from SDV token to other actor token is interesting as an interpretable proxy for importance, this isn't really novel and was previously proposed by CAPO (see citation below).
- It is not fully clear what is the output representation for the auxiliary task of predicting the future trajectories of other agents. As a consequence, $\mathcal{L}_{CE}$ is also unclear.
```
@inproceedings{mcallister2022control,
  title={Control-Aware Prediction Objectives for Autonomous Driving},
  author={Rowan McAllister and Blake Wulfe and Jean Mercat and Logan Ellis and Sergey Levine and Adrien Gaidon},
  booktitle={International Conference on Robotics and Automation (ICRA)},
  year={2022}
}
```
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: I recommend rejection for this paper given that the claims are not validated by the method and experimental setup. 
issues: Please see weaknesses section. I recommend that the authors add a perception module, repeat their experiments, revisit the claims and resubmit to a subsequent conference. While I believe the work is interesting and it is worth continue building on top of, with the experiments as they stand I don't believe readers can take any valuable conclusion.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents PlanT, a transformer architecture for motion planning. Unlike most models which reason according to rasterized BEV scene representations, PlanT leverages an object-centric input representation and simple route encoding. The model is trained similar to prior works using behavior cloning with supervision obtained from a rules-based expert with access to privileged information. The authors claim PlanT's architecture and input representation leads to improved planning performance and more explainable outputs, since the transformer's attention weights provide a measure of importance of each object in the scene. The authors provide a set of comprehensive experiments in the CARLA simulator to support these claims. 
strengths_and_weaknesses: **Strengths**
- **Clear presentation:** The paper is very well written and easy to follow.
- **Very thorough experiments:** I really appreciate the scientific rigour in the experimental analysis and how the authors control for each change between the models. I appreciate the authors comparing against PlanCNN and found the analysis in Table (2a) very well done as it shows exactly where the performance improvements of the model come from. 
- **Simple yet effective model:** The proposed model is quite simple, yet is convincingly shown to be the best performing model. The qualitative difference between the explainability visualization of PlanCNN and PlanT shown in the video is apparent and helps support the claim that PlanT is a more explainable model. 

**Weaknesses**
- **Assumes perfect perception**: As mentioned in the limitations, this paper only considers the problem of planning and assumes perfect perception. While I understand the desire to reduce the scope of the problem, I believe the experimental section or supplementary should show at least one experiment with end-to-end results (from sensor inputs). This would help provide a sense of what the gap is between using GT inputs vs. noisy perception.  
- **Simplified route representation:** While it was an interesting contribution to the paper that a simplified route is sufficient for state-of-the-art performance, this seems like a limitation for a real-world setting. How would this representation support a discretionary lane change (one that isn't needed to follow a route, but that the planner performs for a safer and/or faster driving)? What is the intuition for why all lane information is not helpful for better prediction of the other actors?  
- **Missing connection to transformers for motion-forecasting:** While the end objective is clearly different, the problems of motion forecasting and motion planning share many similarities. Therefore, I think it could be beneficial to mention the connections to the many recent works on leveraging transformers for motion-forecasting and highlight the differences in this architecture. I find that the proposed architecture has similarities with SceneTransformer [1], which also leverages a Transformer to predict the future states of each actor - I believe the key differences between the two are the end objective, the architectural details (cross-attention) and the route representation used in this paper. Additionally, past works in transformers for motion-forecasting [2] have also visualized attention weights to understand the significance actors have on each other (albeit for the motion forecasting task), which may be worth mentioning in the related works. 
- **Only tested in simulation and reliance on a rules-based expert:** This was also mentioned in the paper's limitations, but I'm curious how the authors would imagine using this system in the real-world? Since the model is data hungry, it seems like it would be most feasible to train in simulation and try to transfer the model to the real-world? If that is the case, why leverage an expert and not consider other training schemes like RL? When using the same field of view, it seems like Roach and PlanCNN are within performance (accounting for the confidence intervals). Do you think this model could be trained or fine-tuned with RL in the future to outperform the expert even further? 

References: 
- [1] Ngiam et al., Scene Transformer: A unified architecture for predicting multiple agent trajectories, ICLR 2022
- [2] Li et al., End-to-end Contextual Perception and Prediction with Interaction Transformer, IROS 2020
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I am on the fence for this paper but leaning towards accept. On the one hand, I think this method has limited novelty (when considered in the context of recent transformers for motion forecasting) and has several limitations (most of which are  mentioned in the paper's limitations section) which would need to be addressed before it could be deployed in a real-world setting. However, many of these simplifying assumptions are common in the research community and I find the experimental analysis of this paper very thorough and the detailed ablations and experiments insightful. Additionally, while there are similarities with prior work, applying the transformer to object-centric representations for motion planning is novel, as far as I know. Therefore, since the technical quality is high, the experiments are insightful and the contributions could provide value to the community, I would lean towards acceptance. 
issues: See weaknesses section. 

Some additional minor comments: 
- I personally find the title of this paper misleading. I think it would be more accurate to include "transformer" and "object-centric" as those highlight the key contributions of the paper. 
- Re the auxiliary prediction loss the authors say, "However, since this is a self-supervised objective, it can be incorporated without additional annotation costs". I find this statement misleading as it is only true in the simulated setting and may not necessarily be true in a real-world setting. It's not clear for real-world data whether one would use the outputs from the perception system (which would be noisy, but not require annotation) or GT human labels (which would be more accurate, but costly to annotate).
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper designs a novel object-level scene representation and proposes a transformer based framework to predict the set of waypoints given the route. The framework outperforms other baselines and is more efficient. The attention mechanism also improves the explain explainability of the method.
strengths_and_weaknesses: Strengths:
1.Transform the scene representation from pixel level to object level which is closer to human perception.
2.Regarding the vehicles and routes as a sequence and utilize transformer to predict the waypoints, which is a novel and reasonable combination.
3.The visualization of the attention score over other vehicles improves the explainability of the decision making process.
4.The attached videos clearly demonstrates the effectiveness of the framework.

Weaknesses:
The framework only considers the controlling process but ignore the perception process, which greatly simplifies the problem. I hope that the future work would take perception into consideration and construct a complete framework.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper designs a novel object-level scene representation and proposes a transformer based framework to predict the set of waypoints given the route. The authors transform the scene representation from pixel level to object level which is closer to human perception by regarding the vehicles and routes as a sequence and utilize transformer to predict the waypoints, which is a novel and reasonable combination.  The framework outperforms other baselines and is more efficient. The attention mechanism also improves the explain explainability of the method. However, the framework only considers the controlling process but ignore the perception process, which greatly simplifies the problem.
issues: Some of the training details like the illustration of the input representations and training strategy should be replaced to the body part.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a transformer-based approach for planning (PlanT) in the context of autonomous driving that takes as input a sparse object-based representation of the environment (in contrast to the denser HD maps commonly used in practice). The approach is demonstrated in challenging simulated autonomous driving scenarios and surpasses prior methods (many based on denser input representations) across several benchmark driving metrics.
strengths_and_weaknesses: # Strengths

- The technical approach, as I understand it, seems sound. The idea of using a purely object-based representation as input to a learned planner seems neat, and the ability to assign some scores to objects related to their "relevance" seems interesting.

- The fact that the planner here takes a *sparse* representation is particularly interesting, as this requires significantly less data than HD maps currently in use (making it also potentially easier to process at high rates).

- The supplemental video was **very** helpful in understanding the system and visualizing the output.

- The experimental evaluation is very thorough, and the detailed appendices are appreciated.

# Weaknesses
(I am combining weaknesses here with issues/clarifications)

- The technical exposition in Sec 3 is very dense. I believe this is in part due to the fact that essentially the entire system is described verbally. I could see this making it difficult for a practitioner to attempt to reproduce this method without access to the code. I would encourage the authors to try to define concisely (even at a high-level) the mathematical model captured in part by Figure 1. In particular, with the current presentation, it is tremendously difficult to visualize how the verbally described components fit together (even with the aid of Fig. 2), or determine any sort of relative importance (i.e. what are really the "main" ideas versus auxiliary components). It seems that almost everything is essentially given the same "priority" in the current presentation. In the absence of such a mathematical model, it would help to give an overview figure, perhaps one similar to Fig. 2, but to provide references to the sections in text where components are discussed. The pair of eq (1) with Fig 2 is a great example of this working successfully. I particularly appreciated that the different loss terms were cross references, and the choice of notation here helped convey that. I wonder if a similar approach could be taken for the rest of the system?

- Strictly speaking, is it fair to call the output of the system "explainable"? I am not an expert in explainable AI, but while the object attention weights visualized in the supplemental video for PlanT do look more "stable" than PlanCNN, it was rarely obvious to me why PlanT highlighted particular car as "relevant" (except in the event of the obvious interpretation that the most relevant car was simply the one immediately in front of the agent). Independent of this, the ability to provide these attention weights seems interesting to me, and does (I believe) help visualize or interpret the internals of the model, compared to a black-box model.

- The downside of the sparseness of the object-based representation in practice is that it may be more error prone than an HD map, for the simple reason that consistency of the (denser) information encoded in the HD map can be used to reason about e.g. outliers. Another natural issue is that the object-based map will typically rely crucially on the output of learned perception models, which can produce spurious detections or misclassifications. The limitations section already mentions that object states were extracted from the simulator directly (and it is reasonable in this work to focus on planning specifically), however, I would suggest discussing the issue of robustness, particularly with respect to spurious or missed object detections and misclassifications. In principle, this may be something that could be tested in simulation.

- Minor, but one of the stated contributions is that "we demonstrate that a simple object-level representation is sufficient to encode all the information relevant for planning in urban driving environments." This seems like far too broad a claim to be made without more thorough evaluation in real environments.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The contribution here (a transformer-based planning approach that processes a parsimonious object-based representation for autonomous driving) seems relevant and novel to me. The paper overall is good, but there are some clarity issues with the technical presentation; this is where it could be improved the most, in my opinion. The evaluation is thorough and clear.
issues: See Weaknesses (reproduced here)

- The technical exposition in Sec 3 is very dense. I believe this is in part due to the fact that essentially the entire system is described verbally. I could see this making it difficult for a practitioner to attempt to reproduce this method without access to the code. I would encourage the authors to try to define concisely (even at a high-level) the mathematical model captured in part by Figure 1. In particular, with the current presentation, it is tremendously difficult to visualize how the verbally described components fit together (even with the aid of Fig. 2), or determine any sort of relative importance (i.e. what are really the "main" ideas versus auxiliary components). It seems that almost everything is essentially given the same "priority" in the current presentation. In the absence of such a mathematical model, it would help to give an overview figure, perhaps one similar to Fig. 2, but to provide references to the sections in text where components are discussed. The pair of eq (1) with Fig 2 is a great example of this working successfully. I particularly appreciated that the different loss terms were cross references, and the choice of notation here helped convey that. I wonder if a similar approach could be taken for the rest of the system?

- Strictly speaking, is it fair to call the output of the system "explainable"? I am not an expert in explainable AI, but while the object attention weights visualized in the supplemental video for PlanT do look more "stable" than PlanCNN, it was rarely obvious to me why PlanT highlighted particular car as "relevant" (except in the event of the obvious interpretation that the most relevant car was simply the one immediately in front of the agent). Independent of this, the ability to provide these attention weights seems interesting to me, and does (I believe) help visualize or interpret the internals of the model, compared to a black-box model.

- The downside of the sparseness of the object-based representation in practice is that it may be more error prone than an HD map, for the simple reason that consistency of the (denser) information encoded in the HD map can be used to reason about e.g. outliers. Another natural issue is that the object-based map will typically rely crucially on the output of learned perception models, which can produce spurious detections or misclassifications. The limitations section already mentions that object states were extracted from the simulator directly (and it is reasonable in this work to focus on planning specifically), however, I would suggest discussing the issue of robustness, particularly with respect to spurious or missed object detections and misclassifications. In principle, this may be something that could be tested in simulation.

- Minor, but one of the stated contributions is that "we demonstrate that a simple object-level representation is sufficient to encode all the information relevant for planning in urban driving environments." This seems like far too broad a claim to be made without more thorough evaluation in real environments.
review_title: null

Meta-review: metareview: The paper presents an end-to-end approach for planning that uses a transformer architecture to predict future waypoints. The paper lays emphasis on sparse object-centric representations and further attempts to extract explanations based on attention scores learned by the transformer architecture. 

The reviewers find the paper well motivated and clearly written and the approach is simple 
and easy to understand. The analysis of attention score correlates with human intuition in driving scenarios. However, the reviews express the concern that this work assumes perfect perception and hence the conclusion that object centric representations are enough to encode all relevant information (say in relation to approaches that take BEV as input) is strong and may not be well substantiated by the results in this paper. Further, the assertion related to the sufficiency of the driving route (in contrast to the whole map) needs further analysis to ascertain if the effect is observed due to the characteristics of the simulation setup.

During the rebuttal phase, the authors addressed a central issue of assuming perfect perception and provided results with an integrated off the shelf perception module. Further, the technical exposition of the proposed architecture was strengthened with inclusion of details on the waypoint prediction and auxiliary loss which improve the paper.  In relation to the assertion that the driving route (as opposed to the whole map) is sufficient for planning, the authors concur that the in general the route may not be encoding all the context needed for planning. 

Based on the inputs from reviewers, I observe that the paper attempts to tackle several questions associated with learning planners for autonomous driving: whether object centric representations are better or worse than BEV-context techniques, the need for explainability at object level, how can transformer-inspired architectures can be used in this setting and how well do they work. Given the breadth of issues, it is challenging to clearly articulate the technical gap that is finally filled with this paper. Hence, I would strongly urge the authors to explicitly state the precise technical gap their work is trying to fill (somewhere between sections 2 and 3) which would help the reader to better situate the contribution of this work. 



confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)