Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a system called LCTGen for generating traffic scenarios from language descriptions. The system first uses an Interpreter module to convert a textual description of a traffic scenario into a compact, structured format. This is used by a Retrieval module to sample a map region that fits the structured scenario representation. The structured representation and map are given to a Generator module, which is a transformer model that outputs a scenario. The authors use the Waymo Open Dataset to evaluate scene reconstruction quality of the Generator module and find that it outperforms two baselines - TrafficGen and MotionCLIP. The authors then evaluate the language-conditioning aspect on two small datasets (NHTSA crash reports and Attribute Description dataset) using a human study. Finally, the authors show that LCTGen can also be used to edit traffic scenarios via commands expressed in natural language.
review: Strengths:
- The proposed approach enables use of real-world traffic incident datasets, which might be stored in textual format rather than geometric format. This can allow generation of simulated traffic scenarios that better reflect the type of incidents that actually occur in real life.
- The approach leverages the abilities of LLMs such as GPT-4 which excel at textual conversion tasks.
- The proposed traffic scenario generative model outperforms baselines.
- The human study shows that users find scenarios generated by LCTGen to better match the input text descriptions.

Weaknesses:
- The evaluation datasets for language-conditioned simulation appear to be small scale (40 cases for one, 38 cases for the other).
- The authors propose use of an LLM to interpret textual scenarios as a key component of their system. However, there appears to be no evaluation of that component in isolation.

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This paper proposes an interesting approach for language conditioned traffic scenario generation for testing self-driving in simulation. Experiments show strong performance of the proposed approach compared to baselines. However, additional analysis is needed for LLM task in the Interpreter module, as well as qualitative description of what information language adds that can explain the improved performance.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper introduces LCTGen, a generative model combining LLMs and transformer based decoder to generate controllable and realistic traffic scenarios. It primarily consists of an interpreter and a generator, where interpreter converts the query to a structured representation of the scene consisting of map and agents scecific information using GPT-4. It follows a Chain of Thought promoting to summarize the scenario using GPT-4 and plan agent by agent to generate structured representation of the scene. Generator uses structured representation and sampled map to generate realistic traffic scenarios that matches the user's specifications. Results demonstrate that the proposed method outperforms the existing baselines (TrafficGen and MotionCLIP) for language conditioned generations and simulations, and is evaluated up with human studies.
review: Strengths:

1. Most of the paper is well written, and is easy to read. 
2. The idea of leveraging LLMs for realizing, and controlling complex traffic scenarios is quite novel, and interesting.
It has immense potential to generate and augment dataset with rare and safety critical scenarios. 
3. The baselines are evaluated on scene reconstruction, and language conditioned simulation, and the results are 
backed up with human studies.

Weaknesses:

- While overall the paper is easy to read, there are certain sections which aren't easy to comprehend as follows:
 - Abstract and Introduction: These sections don't holistically provide the motivation, and future scope and applications of this work, be it out of distribution generalization or reducing sim to real gap. 
- Few modules aren't easy to understand from the current version. For instance: the map dataset generation, association and feasibility of input text with the map in the map dataset.
- there are no details of the input format of map encoder, and human study involved.
- No qualitative results shown for other 2 baselines: MOTIONCLIP, TrafficGen

For language conditioned generation, While the current method doesn't seem to capture the infrastructure information like traffic signs, it would be interesting to see if the traffic light turns red, how does the behavior of the agents change at the intersection.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I would be happy to increase my rating, if the issues are addressed in the updated draft.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a method LCTGen that can leverage LLM to generate traffic scenes. It has 3 components: interpreter, generator and encoder. The interpreter convert natural language into a structured representation and then generator takes the structured representation and map to generate realistic traffic. Experimental results show that LCTGen can generate realistic and interesting traffic scenarios and this technique can be applied to applications such as instructional traffic scenario editing and controllable driving policy evaluation.
review: This paper is well written and the idea is novel. Experimental results show much better results than baseline methods.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Using language to generate traffic scene is an interesting idea. It is much more efficient than collecting real traffic pattern or manually generate traffic scenes. I would recommend accepting this paper.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This paper proposes a novel method to generate traffic scenes conditioned on languages. Their method contains three components, a GPT-4 based Interpreter that converts language into features, a Retrieval that retrieves suitable maps based on feature similarity and a Generator network that generates traffic agents based on maps and previous features. Experimental results show that their method outperforms previous baselines, and generates traffic scenes that are both more realistic and aligned with the language. 
review: Strength

1.	The method that uses GPT-4 to generate traffic scenes is interesting and novel.
2.	The performance improvement is decent.
3.	The method can be used for language-conditioned traffic scene editing, which is never shown by previous methods before. 
4.	The paper writes well

Weakness

1.	It would be interesting to include an ablation experiment to study whether open-sourced LLMs can replace GPT-4 in the system.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I think the method shows nice novelty and performance. In addition, I like the usage of language-conditioned traffic scene editing, it is quite novel. I think the method is potentially important for autonomous driving and deserves a presentation at the conference.
review_title: null

Meta-review: The paper introduces LCTGen, a novel system for generating realistic traffic scenarios from natural language descriptions. It includes an Interpreter module for converting text into structured representations, a Generator module utilizing both structured input and map data to create coherent scenarios. The system outperforms existing benchmarks in language-conditioned scenario generation and scene quality, as demonstrated by evaluations on various datasets including Waymo Open Dataset and human studies. LCTGen's versatility extends to scenario editing through natural language commands and controllable driving policy evaluation, showcasing its potential for practical applications.

The reviewers pointed out the paper's strengths in its innovative utilization of LLMs for scenario generation, demonstrating performance improvements over baselines, and the exploration of novel applications such as language-conditioned editing. Additionally, the paper's high-quality writing contributes to its overall impact and accessibility.

The main weaknesses of the paper pointed out by the reviewers include that the evaluation datasets are small, the LLM component is not evaluated in isolation, and there is unclear or missing information in certain sections of the paper.

During the rebuttal phase, the authors provided informative new ablation study on different LLMs. 

We recommend accepting this paper as poster. The authors should include the ablation study on different LLMs and address other weaknesses in the camera ready version.
Decision: Accept (Poster)