Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper proposes relaxing the collision-free requirement of motion planning via the introduction of "Language-Conditioned Path Planning" (LAPP). In LAPP, the goal is to generate a joint-space path where semantic information (as a language command) decides what collisions are allowed. The paper proposes Language-Conditioned Collision Function (LACO), which takes in a single-view image, a robot configuration and a language command and returns a collision probability.

Trained with cross-entropy loss, the learning pipeline tokenizes the image, language and robot configuration by a CLIP vision encoder, CLIP language model and MLP respectively. The tokens are passed into a transformer and the output is averaged to provide collision probability. The collision function is trained with a large simulation dataset, fine-tuned with real-world data. Ablation results compare the LACO's performance and the function is also integrated into an optimization-based motion planner. 
review: The heart of this paper is the interesting viewpoint that, in many motion planning settings, not all collisions are created equal. In allowing some collisions, the planner can solve previously impossible or very difficult tasks. The paper focuses on the setting where the collision information is provided via language conditioning (LAPP). There is a wide range of semantic information that one could imagine extracting from language, i.e. that is okay to partially collide with an object like a shower curtain since the curtain will confirm, that colliding with a full glass of water is more disastrous than an empty glass, or that for some objects it is okay to slightly bump the object as long as that contact is minor. This is exciting and I believe future work could build on the idea of LAPP to explore these types of problems. 

As a step towards this, the paper proposes the more narrow, although very well-defined, application of using language to specify which objects to collision check against (LACO). LACO's methodology is well-described and it leverages state-of-the-art work in encoding images and language. The specification is such that it could be easily plugged into a motion planner (as the paper demonstrates). 

One major question is that it is not clear what the syntax/form/phrasing of the language command is. Are the descriptions given in Fig. 5 examples of the language command or is the language command the list of objects? Does the syntax impact performance? Additionally, LACO outputs collision probability. While this is converted to a binary value (I believe through thresholding), it would be interesting to consider treating it as a cost in the context of an optimization-based motion planner. 

The experiments evaluate LACO as compared to non-language conditioned planners and conduct several ablation studies. A variant of TrajOpt where LACO is used as the collision-checker is applied to a reaching and pushing task. For the pushing task, there are only three instances and the results are the same with and without conditioning on language, which does not support the claim that conditioning on language is useful in this task. Additionally, how is the downstream task success defined? For the failures, does this mean that there was an undesirable collision, no path was found or that the robot failed to reach the target? 

The paper discusses several interesting limitations that serve as great areas of future work. In general, the paper is well-written and easy to follow. The related work seems sufficient. 

A few more minor questions and comments: 
- Does the single-view image include the robot? Sec 3.2 states that the environment configuration is agnostic to the robot joint states - is the robot masked out of the image? 
- Sec 4.2 mentions that state is respected to (x, y) position of the end effector. One interpretation (that seems unlikely) is that only the end effector is considered for collisions, not the entire configuration is the arm. Since I think this is the wrong conclusion, what does state mean in this case? Is it that, in collecting real world data, the objects and end effector position were only varied in (x, y)? 
- Since SceneCollisionNet is trained on a different data set, its unclear if the comparison is fair. However, I do not see this as a major concern, since SceneCollisionNet is not solving the same problem as LACO and the comparison is only one small part of the results. 
- For the simulation results, is the Built-In collision checker used as ground truth? 
- Sec 3.3, Line 134: "implement LAPP using a sampling-based method, LAPP-TrajOpt [7]." -> "implement LAPP using an optimization-based method, LAPP-TrajOpt [7]."
- Sec 3.3, Line 135: "additionally integrate a sampling base method, LAPP-RRT* [6]" -> "additionally integrate a sampling-based method, LAPP-RRT* [6]"
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper presents an interesting take on modulating what it means for a path to be collision-free. The method incorporates advances from computer vision and natural language and incorporates them into a motion planning setting. There are extensive ablation studies and motion planning experiments. There are a number of request clarifications that would greatly strengthen the paper. However, I believe these could be feasibly addressed during the rebuttal/revision period. 

[After the rebuttal, I would still recommend the paper for acceptance. I have also added a comment below]
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper proposes a path planning algorithm that includes a collision function based on language conditions. The function calculates a collision cost based on the given image, robot state, and language prompt. This collision function, called the language conditioned collision function, outputs a collision score. By incorporating this function into a sampling-based planning algorithm, the robot can navigate to the goal while considering desired or undesired collisions. The authors train the network using both simulation and real-world data. The paper also discusses the advantages over existing collision checkers and presents some ablations.
review: The authors introduce a new robotics domain where certain collisions are desired while others are not. They propose the use of language prompts to express human ideas effectively. The design of a language conditioned collision checker is novel and inspiring. The writing in the paper is clear, and the methods are well-presented. Considering that language conditioned planning is a new problem, it is acceptable that there isn't extensive comparison demonstrated in the paper. The limitation section highlights some drawbacks of the current methodology and suggests that making the language prompts more general would enhance the significance of the work.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper provides valuable insights into the concept that collisions can be desirable in robot manipulation tasks. To differentiate between desired and undesired collisions, the paper introduces a language-conditioned collision checker. By utilizing language prompts, the robot can plan a path that includes acceptable collisions with specific objects. Real-world demonstrations validate the effectiveness of the method. Overall, the paper showcases an implementation that contributes to the development of higher-level intelligent robots.

Update: Thanks for the efforts of the authors. My concern is addressed. I recommend accepting this paper.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This paper presents a novel collision checking function that can seamlessly integrated with existing planners. The novelty of this function is that it considers cases where collision with certain objects is allowed. Acceptable collisions are specified as a language prompt. Then proposed collision checking function, called LACO, reasons about collision using a single image, a language prompt, and the current robot configuration.




review: Strengths:

+ the considered problem of allowing collisions with certain semantic objects is quite interesting and practical.

+ The proposed method can be easily integrated with existing planners.

+ The provided experiments nicely demonstrate the proposed method.

Weakness

- The technical approach needs to be motivated more. Existing semantic planners can possibly be used to address similar problems without the high computational cost of designing neural network-based collision functions.


robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall the paper addresses an interesting problem and the proposed algorithm can be a valuable contribution in the field of robot planning.
review_title: null

Meta-review: This paper suggests a new approach called "Language-Conditioned Path Planning" (LAPP) that relaxes the need for collision-free motion planning. LAPP works by allowing semantic information, like language commands, to determine which collisions are acceptable as it generates a path in the joint space. The paper presents an interesting, novel approach backed by extensive ablation studies and motion planning experiments.
Decision: Accept (Poster)