Language-Conditioned Path Planning
Authors: Amber Xie
Youngwoon Lee
Pieter Abbeel
Stephen James
Venue: CoRL 2023 Poster (2023)
Keywords: Robotic Manipulation
Path Planning
Collision Avoidance
Learned Collision Function

Abstract: Contact is at the core of robotic manipulation. At times, it is desired (e.g. manipulation and grasping), and at times, it is harmful (e.g. when avoiding obstacles). However, traditional path planning algorithms focus solely on collision-free paths, limiting their applicability in contact-rich tasks. To address this limitation, we propose the domain of Language-Conditioned Path Planning, where contact-awareness is incorporated into the path planning problem. As a first step in this domain, we propose Language-Conditioned Collision Functions (LACO), a novel approach that learns a collision function using only a single-view image, language prompt, and robot configuration. LACO predicts collisions between the robot and the environment, enabling flexible, conditional path planning without the need for manual object annotations, point cloud data, or ground-truth object meshes. In both simulation and the real world, we demonstrate that LACO can facilitate complex, nuanced path plans that allow for interaction with objects that are safe to collide, rather than prohibiting any collision.

Language-Conditioned Path Planning
Amber Xie
Youngwoon Lee'
Pieter Abbeel
Stephen James?
'University of California, Berkeley
Dyson Robot Learning Lab
https://amberr:t gitgtbtthhhu iotlipp//ppp//
Abstract: Contact is at the core of robotic manipulation. At times, it is desired (e.g. manipulation and grasping), and at times, it is harmful (e.g. when avoiding obsta- cles). However, traditional path planning algorithms focus solely on collision-free paths, limiting their applicability contact-rich tasks. To address this limitation, we propose the domain of Language-Conditoon awareness incorporated into the path planning problem. As a first step in this domain, we propose Iinguae--Conitionn approach that learns collision function using only single-view image, language prompt, and robot configuration. LACO predicts collisions between the robot and the environment, enabling flexible, conditional path planning without the need for manual object annotations, point cloud data, or ground-trus -ruth object meshes. In both simulation and the real world, we demonstrate that LACOO can facilitate complex, nuanced path plans that allow for interaction with objects that are safe collide, rather than prohibiting any collision.
Path Planning, where contact-
Collision Functions (LACO), novel
Keywords: Robotic Manipulation, Path Planning, Collision Avoidance, Learned Collision Function
Introduction
Collision checking is fundamental aspect of path plan-
Collision free
"Can collide with toy"
ning in robotics [1, 2, 3, 4, 5, 6. 7]. aiming find path between initial and target robot configurations that avoids collisions with the environment. However, tra- ditional collision-free path planning approaches fall short in scenarios where contact with the environment necessary, such when manipulating objects or in- teracting with the surroundings. such cases, the strict "collision-free constraint becomes impractical and in- hibits the robot's ability to perform tasks effectively.
(a) w/o language
(b) w/ language
Figure I: Language-conditioned path plan- ning enables finding path, e.g., toward cup, with acceptable collisions, e.g., a plush toy (right). whereas typical path planning fails at finding a collision-fii path (left).
Traditional approaches for enabling contact in path planning [8, 9] often require manual adjustments, such as disabling collision checking for specific objects. However, these approaches rely access to object
state information, ground-truth object meshes, or extensive engineering efforts for each execution. This poses significant challenges, particularly in vision-based contact-rich robotic manipulation tasks.
To overcome this limitation, we propose the domain Language-Condition (LAPP), which integrates planning is not solely concerned with avoiding collisions but also incorporates the ability to make informed decisions about contact with the environment. This enables robots perform complex manipulation tasks that involve controlled interactions, such as holding a cup or opening door. Figure provides an illustration of typical scenario where robot encounters multiple obstacles and needs to interact with the environment.
Path Planning into the path planning problem. this domain, path
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

To facilitate flexible and adaptive contact-aware path planning, we propose Collision Functions (LACO) as an initial step the language-conditioned path planning domain. LACO learns to predict a collision based on a single-view image, language prompt, and robot configuration. By predicting collisions between the robot and the environment modulated by the language prompt, LACO enables the generation of path plans that can handle both desired and controlled collisions without requiring manual object annotations, point cloud data, or ground-truth object meshes. This approach empowers robots to interact with objects that are safe to collide, rather than rigidly avoiding all collisions.
Conditioned
In summary, our main contributions are threefold:
We propose a novel domain of semantic language commands to enhance the robot's understanding of how and what to interact with in the environment. By fusing language instructions, we enable more intelligent and context-aware planning.
Conditionnd Path Planning (LAPP) that integrates
path planning, we introduce Language-Conditionee Conditioned Collision Functions (LACO), a collision function that incorporates language prompts to modulate collision predictions. LACO utilizes only a single-view camera, climinating the need for object states or point clouds. This allows for easier application in real- weal-world scenarios and facilitates zero-shot generalization new language commands.
To enable
We provide comprehensive demonstrations of LACO's effectiveness in various path planning tasks. Our experiments include simulations and real-world scenarios, highlighting the practicality and robustness of our language-conditiong collision function.
2 Related Work
Path planning [1, 2, 3, 4, 5, 6, 7] finds a collision-free path between initial and target task con- figurations- - representing robot states and potentially environment states - by querying a collision function with states along a path. However, these collision-free path planning methods struggle handling scenarios where collisions with the environment desired. this paper, we propose lunguage-conditioned path planning, which finds contact-aware path following a language prompt.
Semantic planning [10, 11, 12] builds a semantic map with specifically-desige pipeline, consisting of, e.g., object detection, segmentation, and keypoint estimation modules. The semantic map is then used to find a collision-free path for navigation. In contrast, we propose an end-to-end language-connditonee collision function, which enables contact-aware path planning.
perception
The recent success in large language models enables training multi-task robotic policies guided by language instruction [13, 14, 15] and high-level planning using large language model [16, 17]. These language-conditionee "task" planning/ / solving approaches differ from our proposed language- conditioned "path" planning domain in that they directly solve specific set of tasks while our language-conditione conditioned path planning is agnostic to downstream tasks and can be used as a building block for many robotics tasks.
A collision function is fundamental component of path planning in robotics. However, the collision function is assumed to be manually modelled or computed using state estimation. Instead of hand- engineering a collision function, many recent work have learned end-to-end motion planners [18, 19, 20] or collision functions [21, 22, 23] from synthetically generated data in simulation. To understand 3D configuration of environments, these approaches use point clouds to represent the scenes. this paper, we propose to learn a collision function from only single camera input, not requiring depth sensing nor precise camera calibration, which makes our method easily applied to many real-world applications. More importantly, which objects to collide or not to collide, allowing acceptable or desired collisions for path planning.
learned collision function conditions language describing

---

language-condiitione collision function (LACO)
Langunge-condittoned path planning (LAPP)
initial
candidate paths
target
CLIP
Encoder
Prediction Net
collision
Traosformet
"pringles"
Lang Encoder
x
LACO
"pringles"
s=(0, mnnd
MLP
0.2
0&8
collision!
Figure 2: A language-conditioned collision function (LACO), C(o, s,(), predicts whether robot state collides with objects other than collidable objects described in language prompt in scene o, e.g., C(o,s, s, 'pringles') 1. To find language-conditoone path plan, a path planning algorithm asks LACO whether any waypoint 8; of path collides with objects except for the ones described in L.
Language-Conditioned Path Planning
In cluttered real-world environment, collision-free path can be highly sub-optimal or impossible to find. We introduce the problem domain of language-conditione path planning, which extends traditional path planning to allow safe or desired collisions in path via language, in Section 3.1. Furthermore, we present proof-of-concept framework for language-conditioned path planning that first learns a language-conditine function for language-conditine conditioned path planning (Section 3.3).
collision function (Section 3.2) and leverages this learned collision
3.1 Londuage-Condittoned Path Planning (LAPP)
In robotics, a path planning problem about finding a connected, collision-free path of robot configurations (i.c. waypoints), p = (so, $1, sg), starting from an initial configuration so to final configuration 8, such that every configuration in the path has no collision with the environment, Vs, P. Co(,,. $t)
o, and 8; denote the environment and robot configurations time respectively, and C(o, denotes collision function that outputs if the robot configuration has collision with the environment and O, otherwise.
In this paper, we propose a the strict collision constraint in path planning so that path planning can manage safe or desired contacts with the world, especially guided by language. LAPP can formulated as finding a path p between configurations, Sp and Sy, such that Vs. p. C(o,, St, () 0, where / language prompt modulating what needs to be considered acceptable desired collisions. For example, language prompt can be "a robot illustrated in Figure 1, or "a robot can grasp mug" to support contact-rich tasks.
Conditioned Path Plamning (LAPP) problem, which relieves
collide with plush toys" to specify safe-to-collide objects, as
3.2 Conduage- Collision Function (LACO)
For language-conditioned path planning. path planning algorithm should understand which collisions are acceptable and which must be avoided described in language. In this paper, we address this problem by adapting a collision function C(o, s) into a Language-Condinng Condiitooed Collision Function (LACO) C'o,, s, I). which takes a language prompt into account. Specificallyy, LACO learns a collision function C(o, 8, 8,),, where o is a single-view image observation of the environment, queried robot joint state, and is language instruction corresponding objects that allow for collisions. Note that does not need to correspond and represents only the environment configuration. Thus, the same can be used across multiple robot configurations

---

CHEEZAI
Mou
(a) Training objects
(b) Held-out objects
(c) Real-world objects
Figure 3: We use ShapeNet objects [27] for our simulated experiments. (a) The training dataset of ShapeNet classes includes: airplane. chair, pot, vessel, laptop. bus, cap, and bottle. (b) The held-out evaluation dataset includes: basket, mug, train, bag. and can. (c) We also perform real-world experiments with YCB objects [28]: Spam, Cheez-it, Pringles, Windex, mustard, and bleach.
We train C(o, s, on dataset D {(o, s,L, y)), where y indicates whether the robot state has any collision in the scene o; and y' indicates whether there a undesirable contact under the language instruction We optimize the cross-entropy losses for language- modaulate collision prediction (target y²) and collision prediction without language conditioning (target y) an auxiliary task:
(o wDw (y C(o, s, s)) + CE (y. Caus(o. s, {))
(1)
where Caxx s,() 1) is additional MLP head attached the last layer of C(o,s,))
To take advantage large vision- language models pretrained on a large corpus [24, 25], LACO uses the vision and language encoders of CL.PP [26] as the backhone networks. As illustrated in Figure we tokenize an input image of size 256 x 256 with the frozen pretrained CLIP VIT encoder and language prompt with the frozen pretrained CLIP language model. We then get 197 visual tokens 20:196 and 197 language tokens 10:196- For robot state single state token z². All these tokens are then fed into 2-layer transformer and the average of the transformer output tokens is used to predict the collision probabilities with two separate MLP heads, C(o, 8,1) and Caux (o, s, I). More hyperparameters are described in Appendix, Table 7.
3-layer MLP to embed it to
3.3 Path Planning using LACO
Finally, language-conditioned path planning can be performed by simply replacing collision checker in any path planning algorithm with LACO. In this paper, implement LAPP using optimization- based method, LAPP-TrajOpt [7]. Whenever LACO needs binary output (collision not), we apply a threshold of 0.5 to the collision probability output of LACO. LACO is flexible to various styles of path planning algorithms, such as sampling-base planning. We use a custom implementation of TrajOpt [7] and the hyperparameters for TrajOpt can found in Appendix, Table 8.
Experiments
In this paper, we introduce ianguage-conditione path planning (LAPP), and present a framework that combines existing path planning algorithms and our proposed language-conditione collision function (LACO) an initial step. Our evaluations twofold: (1) present thorough investigation of LACO's performance in object- and language-level generalization, and (2) we showcase the potential of LACO in the language-conditioned path planning domain.
4.1 Environment Setups
We use the UFACTORY xArm7, a low-cost 7-DOF robotic arm, and an Intel RealSense D435 camera. For the real world, we use 6 YCB objects [28]: Spam, Cheez-it, Pringles, Windex, mustard, and bleach. For simulation, use ShapeNet v2 objects in CoppeliaSim [27, 29], which provides taxonomy of diverse, realistic set of 3D meshes with their labels, as shown in Figure 3.

---

4.2 Data Collection
To train LACO, first collect data in both the sim- ulation and real-world environments.
Simulation dataset. We use PyRep [30] based on CoppeliaSim [31] to synthetically generate diverse, language-annokann dataset simulation. Each scene includes 2-5 randomly chosen objects in random poses on the table. Instead of randomly initializ- ing robot pose, a set of robot poses are generated by the built-in RRT* motion planner [6] for each scene. These smooth trajectories bias the dataset to- ward joint states likely to be queried by a path planner. We use the built-in collision checker for the ground truth collision label y. Next, we sample combinations of 0, .., 1 objects in scene with N objects to generate language annotations (a list of ShapeNet names of the sampled objects) and compute aanguage-oonitione collisions y'. We generated 5000 unique scenes, which consist of unique combinations and positions of objects. Each scene contains about 40 joint states and 10 language annotations.
CHEEBOT
QEEE
Figure 4: The real-world (left) and simulated (right) environments.
Real-world dataset. Learning a collision function from real-world data poses additional challenges: the increased visual complexity and the difficulty of collecting collision data. To address these issues, train LACO dataset collected from domain-randominze twin simulator environment (Figure 4) and then finetune on small real-world dataset. We collect data from 20 real-world scenes and 500 -omain-randomized simulation scenes. For each scene, we extract 20-30 images with domain randomization in simulation and camera perturbations in the real world. Similar to our simulation dataset, we vary the number of objects in each scene from to and vary positions of objects.
4.3 Collision Prediction Results Across Language Conditioning
LACO possesses the ability to be mod- ulated by language. In particular, any number of objects in the scene can be included in the language condition, allowing for flexibility in path plan- ning. This ability is not found in built- in collision checkers or learned col- lision checkers, such as SceneColli- sionNet [21], which are agnostic to desired and undesired collisions.
Table 1: We evaluate the language-condditonne conditioned collision prediction accuracy of LACO in simulation. For reference, we also evaluate SceneCollisionNet [21]. which does not support language conditioning and "Built-In Collision" refers to the ground-truth collision checker.
Accuracy Per Conditioned Objs (%)
Method
2
4
Built-In Collision
100.0
ScemeCollisionNet
67.0
LACO
82.9
78.9
77.18
82.6
72.2
We evaluate the performance of LACO across different number of ob- jects included in the language condition in Table To measure collision prediction accuracy, we sample 10 trajectories with in total N 2000 states, and evaluate 1/x Σ.V Di>1e We find that LACO is robust to different numbers of conditioned objects. though performs best when not conditioned on any objects. In this special case, LACO becomes typical collision checker, without need to understand the semantics of the environment objects.
Even in the unconditional case, our method outperforms SceneCollisionNet [21], a point-cloud-bate based learned collision function. We use the official implementation of SceneCollisionNet, which is trained on a different simulator dataset. This distribution shift may reason for its poor performance in our environment. While our primary contribution presenting new paradigm of collision checking and path planning, the accuracy of our RGB-only method shows promise of collision checking without extensive camera setups and point clouds.
5

---

4.4 Generalization Experiments
One advantage of LACO is its ability to be modulated by language, which is flexible, abstract, and simple. We may expect that with its pretrained vision-lange lainnnla backbone, LACO may generalize to objects and language that are unseen the training dataset.
Generalization to unseen language. We first evaluate generalization unseen instructions of the seen objects. We compare the language-conditioned collision prediction accuracy with the original object name (Default), unseen synonyms of the object (Synonym), complex phrases describing the object (Description), and correct and incorrect colors (Color). For example, collision to "hat" (Default) is evaluated with "beanie" for Synonym, "head-covering accessory" for Description, and "blue hat" for Color. For Color, we also use incorrect colors, e.g., "white hat", which ask LACO to ignore the objects. The exhaustive list of such variations can be found in Appendix, Table 9.
In Table 2, we find strong generalization for Color, where the language references the seen class name. LACO also generalizes to Description, while Syn- onym leads 15.0% accuracy drop. One hypoth- esis that the descriptions, which are typically longer and contain many keywords relating to the
Table 2: We evaluate generalization of LACO to instructions with unseen language.
Default
Synonym
Description
Color
78.9
63.9
71.4
77.0
object, may be more informative than just synonym. Furthermore, short language conditions, like "cap," may even be ambiguous, as "cap" may refer to a bottle cap, hat, or more. This suggests promise in future work of exploring stronger and more descriptive annotations, as we limit our language conditions to ShapeNet class names.
Generalization to unseen objects. We evalu- ate generalization to unseen objects across seen and unseen classes by measuring the collision prediction accuracy for ianguage-conditioni on a single unseen object.
Table 3: Evaluation of LACO's generalization to new objects.
Seen Class
Unseen Class
Seen Object Unseen Object
Unseen Object
78.9
70.6
54.7
LACO achieves comparable collision prediction
accuracy for unseen objects from seen classes, showing its strong generalization due to the pretrained vision encoder. However, LACO shows 24.2% lower accuracy for objects from unseen classes. Unlike the generalization to unseen language alone in Table 2, generalization to the unseen classes is more challenging as it has both class names and objects unseen during training.
4.5 Ablation Studies
Table 4: Ablation on Single and Multi- View Encoders.
Pretrained observation encoder. We investigate the benefit of using pretrained encoder by comparing CLIP pretrained vision encoder and a CLIP vision encoder trained from scratch. Table 4, we find that a pre- trained CL.IP encoder consistently out- performs the one trained from scratch.
Accuracy Per # Conditioned Objs (%)
Method
2 3
4
LACO
82.9
78.9
77.2
82,6
72.2
Finetuning
71.5
64.9
65.14
72.9
68.9
From scratch
82.1
72.7
77.7
75.3
71.1
LACO+ MV
68.8
69.4
76.3
73.4
68.9
Finetuning MV
74,2
66.4
62.6
75.6
68.9
From scratch + MV
66.0
67.9
67.1
78.6
72.7
Multiview camera inputs. To extend to multi-camera RGB observations, we train multi-view MAE to replace the CLIP vision encoder. The multi-view MAE is trained end-to-end to predict image reconstructions of two fixed camera views. Unlike MV-MAE [32] and Multi-MAE [33], we keep the original MAE masking ratio of 80% per each view. When objects are entirely masked out from one view, we find that they can be reconstructed if present in the second view. Sample reconstructions and hyperparameters are included in Appendix C. In Table we find that using pretrained MV-MAE, whether frozen or with finetuning, outperforms training from scratch. However, multi-view feature extraction remains open problem, single-view features lead to stronger predictions.

---

Dataset size. Although Section 4,4 shows the generalization capability LACO, its generalization to unseen classes of objects is This may arise because we are training on a limited dataset of objects.
Table 5: The dataset size ablation examines how perfor- mance varies as we alter the size of the dataset.
Accuracy Per # Conditioned Objs (%)
Dataset Size
0
2
3
As the quality and quantity of 3D assets
50%
82.9
74.2
73.4
78.0
73.6
increases, may expect improved perfor-
80%
81.2
70.0
72.6
72.2
65.8
mance by training more than a limited 100%
82.9
78.9
77.2
82.6
72.2
set of classes. We verify our hypothesis by varying the dataset size in Table 5. The results show that there is an improvement with the increased size of the dataset, but the improvement marginal.
4.6 Condunge- Path Planning Demonstrations
In this section, we showcase three tasks using language-condittoned path planning with LACO:
Reach Target (No Lang): This task resembles traditional path planning task, which aims
reach target joint pose while avoiding obstacles. We do not condition language.
Reach Target (1 Lang): The objective likewise reach target; however, object specified
collidable, allowing for more flexibility in plans.
The objective push an object forward. For this task, collisions in fact desired, showcasing the usefulness of LAPP.
Push Object (1 Lang):
We report the success rates of LAPP in Table 6. Each path planning is considered successful if valid path is found and the path reaches a target or pushes an object without undesirable collisions. The push object task is particularly well-suited for language- conditioned path plamning: TrajOpt can be initialized with trajectory passing through the object and op-
Table 6: We evaluate the success rates of LAPP- Trap-Trajppt on three path planning tasks.
No Lang
Lang
Reach Target
Reach Target Push Object
7/10
8/10
9/10
timize collision constraints with other objects. Reaching targets and pushing objects can also be composed to perform more complex tasks, such as avoiding all obstacles before reaching the object the arm needs to push.
4.7 Real-World Experiments
show real-world trajectories with LAPP-TrajOpt in Figure 5. LAPP-TrajOpt with LACO, pre- trained on simulator data and finetuned with real-world data, is able to find plans in cluttered environments, using the language condition to discover path that would otherwise be regarded as trajectory with collisions.
5 Limitations
While LAPP with LACO offers promising advancements in contact-aware path planning, there several limitations that should be acknowledged:
Lack of environment dynamics. LACO does not explicitly consider enviromment dynamics. Once an object hit, it may react by being pushed or knocked down, potentially affecting the configu- ration and positions of other nearby objects. This restricts the ability of LAPP to handle dynamic environments and may lead to suboptimal or unsafe path plans when objects move significantly.
Limited language prompt scope. In our experiments, the language prompt is limited specifying objects that are desirable safe to collide. While this provides valuable control over contact conditions, the current scope of language prompts may not cover the full range of instructions
7

---

Can collide with Pringles and Cheez-It box.
Seery CHEEZE
CHEEZIT
CHEEZIT
CHEEZIT
collide with Pringles.
Can collide with spam.
Figure 5: We demonstrate successful execution of the LraP-TrajOpt path planning algorithm in the real-robot system. A path the goal, specified by the blue jar, blocked by diverse objects on the table, the robot needs to collide with some object. We inform which object is safe and desired to collide using the language prompts: "pringles, cheezit", "pringles", and "spam", respectively.
or interactions that user may desire. Including wider variety of language instructions and specifications could enhance the versatility and adaptability of LAPP and LACO.
Data generation efforts. The training process for LACO relies on combination of synthetic simulation data and manually collected real-world data. Both require significant human engineer- ing and labeling efforts. Exploring advances 3D asset availability [34] and simulation-to-re techniques [35, 36, 37, 38] could alleviate this limitation and enable more efficient training of LACO.
6 Conclusion
In conclusion, proposed domain of limitations of traditional collision-fiee free path planning in contact-rich robotic manipulation tasks. By integrating contact awareness into path planning, LAPP allows robots to make informed decisions about contact with the environment, enabling them to perform complex manipulation tasks effectively.
Path Planning (LAPP) addresses the
As first step towards LAPP, we propose use Language-Connittonee Collision Functions (LACO), which learns predict collisions based visual inputs, language prompts, and robot configurations. This learned collision function eliminates the need for manual object annotations, point cloud data, or ground-truth object meshes, enabling flexible and adaptive path planning that incorporates both desired and controlled collisions.
Acknowledgments
We would like thank Justin Kerr, Chung Min Kim, Younggyo Seo, and Hao Liu for their insightful advice on leveraging pretrained visual-language models and training transformer models. In addition, thank Philipp Wu for assistance with the real-world experiments and Oleh Rybkin for feedback. This work was funded in part by Darpa RACER, Komatsu, and the BAIR Industrial Consortium.

---

References
[1] M. Overmars. A random approach to motion planning. Technical Report RUU-CS-92-32, Department of Computer Science, Utrecht University, 1992.
[2] L. Kavraki J.-C Latombe. Randomized preprocessing of configuration for fast path planning.
In IEEE International Conference on Robotics and Automation, 1994.
[3] N. M. Amato and Y. Wu. A randomized roadmap method for path and manipulation planning.
In IEEE International Conference on Robotics and Automation, 1996.
[4] S. M. Lavalle. Rapidly-exploring random trees: A new tool for path planning. Technical report, Iowa State University, 1998.
[5] J. J. Kuffner and S. M. LaValle. Rrt-connect: An efficient approach to single-query path planning. In IEEE International Conference on Robotics and Automation, volume 2, pages 995-1001, 2000.
[6] S. Karaman and E. Frazzoli. Sampling- algorithms for optimal motion planning. The International Journal of Robotics Research, 30(7):846-8944 2011.
[7] J. Schulman, J. Ho, A. X. Lee, I. Awwal, H. Bradlow, and P. Abbeel. Finding locally optimal, collision-free trajectories with sequential convex optimization. In Robotics: Science and Systems, 2013.
[8] Burget, A. Hormung, and M. Bennewitz. Whole-body motion planning for manipulation of articulated objects. In IEEE International Conference on Robotics and Automation, pages (666-1662, 2013.
[9] S. Srivastava, E. Fang. L. Riano, Chitnis, Russell, and Abbeel. Combined task and motion planning through extensible glamner-independent interface layer. In IEEE International Conference Robotics and Automation, pages 639-646, 2014.
[10] Y. Kantaros, S. Kalluraya, Q. Jin, and G. J. Pappas. Perception- based temporal logic planning in uncertain semantic maps. IEEE Transactions on Robotics, 38(4):2536225 2556, 2022.
[11] V. Vasilopoulos, G. Pavlakos, S. L. Bowman, J. D. Caporale, K. Daniilidis, J. Pappas, and D. Koditschek. Reactive semantic planning in unexplored semantic environments using deep perceptual feedback. IEEE Robotics and Automation Letters, (33):44554455 2020.
[12] Y. Liang, B. Chen, and S. Song. Ssenav: Confidence-awaree semantic scene completion for visual semantic navigation. In IEEE International Conference on Robotics and Automation, pages 13194-13200. IEFF, 2021.
[13] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipula- tion. In Conference on Robot Learning, 2021.
[14] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference Robot Learning, pages 991-1002, 2021.
[15] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manjunath, Pertsch, Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong. F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. Rt-1: Robotics transformer for real-world control at scale. In Robotics: Science and Systems, 2023.
I. Mordatch, Nachum, C. Parada, J. Peralta, Perez,

---

[16] A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, et al. Do as can, not as say: Grounding language in robotic affordances. In Conference on Robot Learning, 2022.
[17] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. In IEEE International Conference on Robotics and Aulomation, 2023.
[18] A. H. Qureshi, A. Simeonov, M. J. Bency, and M. C. Yip. Motion planning networks. In IEEE International Conference on Robotics and Automation, pages 2118-2122 2124, 2019.
[19] A. H. Qureshi, Y. Miao, A. Simeonov, and M. C. Yip. Motion planning networks: Bridging the gap between dearning-based and classical motion planners. IEFE Transactions Robotics, 37 (1):48-66, 2020.
[20] H. Ha, Xu, and S. Song. Learning decentralized multi-arm motion planner. Conference on Robot Learning, pages 103-114, 2021.
[21] M. Danielczuk, A. Mousavian, C. Eppner, and D. Fox. Object rearrangement using learned implicit collision functions. In /EEE International Conference on Robotics and Automation, pages
2021.
[22] A. Murali, A. Mousavian, C. Eppner, A. Fishman, and D. Fox. Cabinet: Scaling neural collision detection for object rearrangement with procedural scene generation. arXiv preprint arXivv:3304.0 09302, 2023.
[23] D. Son and B. Kim. Local object crop collision network for efficient simulation of non-convex objects in gpu-based simulators. In Robotics: Science and Systems, 2023.
[24] H. Liu, L. Lee, K. Lee, and P. Abbeel. Instraction-folloiingg agents with multimodal transformer. arXiv preprint arXiv:2210.1344 2022.
[25] C. Wang, M. Chai, M. He, D. Chen, and Liao. Clip-nerf: Text-and-image driven manipulation of neural radiance fields. In IEEE Conference Computer Vision and Pattern Recognition, pages 3835-3844 2022.
[26] A. Radford, W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, Sastry, A. Askell, P. Mishkin, J. Clark, al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pages $748-8763. 2021.
[27] A. X. Chang, Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Li, Savarese, M. Savva, S. Song, H. Su, al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:151..03012. 2015.
[28] B. Calli, A. Singh, A. Walsman, Srinivasa, Abbeel, and A. M. Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. In 2015 international conference on advanced robotics (ICAR), pages 510-517. IEEE, 2015.
[29] A. Adeniji, A. Xie, C. Sferrazza, Y. Seo, James, and Abbeel. Language reward modulation for pretraining reinforcement learning. arXiv preprint arXiv:2308. 12270, 2023.
[30] S. James, M. Freese, and A. J. Davison. Pyrep: Bringing v-rep to deep robot learning, arXiv preprint arXiv:1906 176, 2019.
[31] E. Rohmer, S. P. Singh, and M. Freese. V-rep: A versatile and scalable robot simulation framework. In /EEE/RSJ International Conference on Intelligent Robots and Systems, pages 1321-1326, 2013.
[32] Y. Seo, J. Kim, S. James, K. Lee, J. Shin, and P. Abbeel. Multi-view masked world models for visual robotic manipulation. In International Conference on Machine Learning, 2023.
10

---

[33] R. Bachmann, D. Mizrahi, A. Atanov, and A. Zamir. MultiMAE: Multi-modal multi-task
masked autoencoders. In European Conference on Computer Vision, 2022.
[34] M. Deitke, D. Schwenk, Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani,
A. Kembhavi, and A. Farhadi. Objaverse: A universe of annotated 3d objects, 2022.
[35] S. James, A. Davison, and E. Johns. Transferring end-to-end visuomotor control from simulation real world for multi-stage task. Conference on Robot Learning, pages 334-343, PMLR, 2017.
[36] J. Matas, S. James, and A. J. Davison. Sim-to-real reinforcement learning for deformable object manipulation. Conference on Robot Learning, pages 734-743, PMLR, 2018.
[37] James, P. Wohlhart, M. Kalakrishnan, D. Kalashnikov, A. Irpan, J. Ibarz, S. Levine, R. Hadsell, and K. Bousmalis. Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized- to-canonical adaptation networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 12627-12637, 2019.
[38] J. So, A. Xie, S. Jung, J. Edlund, R. Thakker, A. Agha-mohammadi, P. Abbeel, and S. James. Sim-to-real via sim- sim-to-seg: End-to-end off-road autonomous driving without real data. In Conference on Robot Learning, 2022.

---

A Implementation Details
Table 7: LACO hyperparameters.
Hyperparameter
Value
Learning rate Learning rate scheduler
3c-5
cosine decay to 1c-7
Mini-batches
32
Training steps
300000
State tokenizer hidden units
(4096, 4096, 4096)
Prediction net hidden units
(512, 256)
Observation tokenizer/encoder
CLIP B/16
Language tokenizer/encoder
CLIP B/16
Attention layers
4
# Attention heads
16
Token dimension
768
Table 8: TrajOpt hyperparameters.
Hyperparameter
Value
Steps
10 [-0.4, 0.4]
Velocity constraint
140
2
80
0.01
0.75
7
1.1
7
0.5
k
10
ftot
0.0001
Ttal
0.0001
Coo!
0.01
Solver
ECOS
Penalty iterations
Convexify iterations
5
Trust iterations
2
Min. trust box size
0.0001
Language Prompts for Evaluation
Language prompts are included in Table 9,
Table 9: Language prompts for evaluation.
Original Noun
Synonym
Description
planter
plant stand
bin for plants
cap
hat, snapback
head-covering accessory, head-covering article of clothing
boat
sailboat, cruise ship
oceanic vehicle
yachting cap
hat, sailor hat
head-covering accessory
airplane
aircraft, airline
aerial vehicle, object that takes flight
omnibus
vehicle
long vehicle for travel, toy with wheels
bottle
water hottle, bottle container
container for fluids, travel-sized water container
flat cap
hat. beanie
head-covering accessory
laptop computer
electronic device, laptop
device for accessing internet, typing device for work
Colors
red, yellow, blue, purple, pink, gray, black, white
12

---

C
Multi-View MAE
In addition to single-view observations, we also experiment with multi-view observations. Instead of using the pre-trained CLIP encoder, we pretrain a multi-view MAE from scratch on the simulator images. Then, we use multi-view features from the frozen multi-view MAE model for collision prediction.
In particular, we do not apply any special masking strategies. Though recent works in multi-view MAEs [33, 32] have applied such strategies, we find that even basic MAE strategy leads to good reconstruction, even of parts occluded in just one view. For instance, in Figure 6, the blue object is completely masked out of the second view, yet the second view able to successfully reconstruct the object.
We encoder with 2 layers and 16 heads, decoder with 2 layers and 16 heads, token dimension of 768, patch size of 16, and masking ratio of 80%. Our image is preprocessed to be (224, 224), following convention. The learning rate 3 10-5, We add learned embeddings to the tokens for each view.
The result in Table shows that multi-view LACO is worse than single-view LACO. We hypothesize that the use of pre- trained CLIP encoder is crucial for extracting useful features for collision prediction. However, we believe if the multi-view MAE is trained with large web-scale data, it can outperform single-view LACO.
Multi-View MAE Training
z
zºz
MAE
z/
x/
Masked Tokens
L.2 Loss
Figure 6: We pretrain multi-view MAE that trained from scratch on simulator images. k the number of unmasked tokens passed into each view. Note that the blue object is completely masked out of the bottom view, yet able to be reconstructed.
13


Reviews:
[{"review_id": "A0QobrKd93t", "reviewer": "Reviewer_FRPN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a path planning algorithm that includes a collision function based on language conditions. The function calculates a collision cost based on the given image, robot state, and language prompt. This collision function, called the language conditioned collision function, outputs a collision score. By incorporating this function into a sampling-based planning algorithm, the robot can navigate to the goal while considering desired or undesired collisions. The authors train the network using both simulation and real-world data. The paper also discusses the advantages over existing collision checkers and presents some ablations.", "summary_of_recommendation": "The paper provides valuable insights into the concept that collisions can be desirable in robot manipulation tasks. To differentiate between desired and undesired collisions, the paper introduces a language-conditioned collision checker. By utilizing language prompts, the robot can plan a path that includes acceptable collisions with specific objects. Real-world demonstrations validate the effectiveness of the method. Overall, the paper showcases an implementation that contributes to the development of higher-level intelligent robots.\n\nUpdate: Thanks for the efforts of the authors. My concern is addressed. I recommend accepting this paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "There is a concern regarding whether a single image representing the scene is sufficient for the problem. Since geometry information cannot be obtained from the image, the collision prediction by the network may be inaccurate. It would be helpful if the authors could elaborate on whether the accuracy could be improved by incorporating point clouds or multi-view images into the network.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors introduce a new robotics domain where certain collisions are desired while others are not. They propose the use of language prompts to express human ideas effectively. The design of a language conditioned collision checker is novel and inspiring. The writing in the paper is clear, and the methods are well-presented. Considering that language conditioned planning is a new problem, it is acceptable that there isn't extensive comparison demonstrated in the paper. The limitation section highlights some drawbacks of the current methodology and suggests that making the language prompts more general would enhance the significance of the work.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "3lpiUpMrUIr", "reviewer": "Reviewer_CVWw", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a novel collision checking function that can seamlessly integrated with existing planners. The novelty of this function is that it considers cases where collision with certain objects is allowed. Acceptable collisions are specified as a language prompt. Then proposed collision checking function, called LACO, reasons about collision using a single image, a language prompt, and the current robot configuration.\n\n\n\n", "summary_of_recommendation": "Overall the paper addresses an interesting problem and the proposed algorithm can be a valuable contribution in the field of robot planning.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Questions\n\n1) Why a neural network model (LACO) is needed to check for collisions given the current robot configurations? The same can possibly done using existing semantic planners such as [A]-[C]. Such planners reason about the location and semantic labels of nearby objects and based on that they make decisions about what objects should be approached and what objects need to be avoided. \n\n[A] \"Perception-based temporal logic planning in uncertain semantic maps.\" IEEE Transactions on Robotics 38.4 (2022): 2536-2556.\n\n[B]  \"Reactive semantic planning in unexplored semantic environments using deep perceptual feedback.\" IEEE Robotics and Automation Letters 5.3 (2020): 4455-4462\n\n[C] \"Sscnav: Confidence-aware semantic scene completion for visual semantic navigation.\" 2021 IEEE international conference on robotics and automation (ICRA). IEEE, 2021\n\nThe authors do not need to cite these specific papers; however, a discussion on how this work differs from existing semantic planners would be helpful.\n\n2) What do the numbers in Table 6 mean?\n\n3) How does LACO generalizes to unseen environmental conditions? For example, consider seen objects/classes but different lightning conditions. A discussion on this would be helpful.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\n+ the considered problem of allowing collisions with certain semantic objects is quite interesting and practical.\n\n+ The proposed method can be easily integrated with existing planners.\n\n+ The provided experiments nicely demonstrate the proposed method.\n\nWeakness\n\n- The technical approach needs to be motivated more. Existing semantic planners can possibly be used to address similar problems without the high computational cost of designing neural network-based collision functions.\n\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "2tXefZwOwTu", "reviewer": "Reviewer_bXSy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes relaxing the collision-free requirement of motion planning via the introduction of \"Language-Conditioned Path Planning\" (LAPP). In LAPP, the goal is to generate a joint-space path where semantic information (as a language command) decides what collisions are allowed. The paper proposes Language-Conditioned Collision Function (LACO), which takes in a single-view image, a robot configuration and a language command and returns a collision probability.\n\nTrained with cross-entropy loss, the learning pipeline tokenizes the image, language and robot configuration by a CLIP vision encoder, CLIP language model and MLP respectively. The tokens are passed into a transformer and the output is averaged to provide collision probability. The collision function is trained with a large simulation dataset, fine-tuned with real-world data. Ablation results compare the LACO's performance and the function is also integrated into an optimization-based motion planner. ", "summary_of_recommendation": "The paper presents an interesting take on modulating what it means for a path to be collision-free. The method incorporates advances from computer vision and natural language and incorporates them into a motion planning setting. There are extensive ablation studies and motion planning experiments. There are a number of request clarifications that would greatly strengthen the paper. However, I believe these could be feasibly addressed during the rebuttal/revision period. \n\n[After the rebuttal, I would still recommend the paper for acceptance. I have also added a comment below]\n\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- What is the form of the language command?\n- LACO outputs a collision probability. Is this value then thresholded when evaluating performance and when used within a motion planner? (If so, what is that threshold?)\n- What are the typical failure cases for LACO?\n- Sec 4.4 considers generalization to unseen language by considering synonyms and descriptions. How were these synonyms and descriptions sourced?\n- In this context of the reaching task for Sec 4.5, how is conditioning on language helpful? In the current results the performance is the same, with or without language.\n\nI have included a few more minor questions in the \"Review\" section. Additionally, if there are videos of the real world experiments, it would be great to include them in the supplemental. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The heart of this paper is the interesting viewpoint that, in many motion planning settings, not all collisions are created equal. In allowing some collisions, the planner can solve previously impossible or very difficult tasks. The paper focuses on the setting where the collision information is provided via language conditioning (LAPP). There is a wide range of semantic information that one could imagine extracting from language, i.e. that is okay to partially collide with an object like a shower curtain since the curtain will confirm, that colliding with a full glass of water is more disastrous than an empty glass, or that for some objects it is okay to slightly bump the object as long as that contact is minor. This is exciting and I believe future work could build on the idea of LAPP to explore these types of problems. \n\nAs a step towards this, the paper proposes the more narrow, although very well-defined, application of using language to specify which objects to collision check against (LACO). LACO's methodology is well-described and it leverages state-of-the-art work in encoding images and language. The specification is such that it could be easily plugged into a motion planner (as the paper demonstrates). \n\nOne major question is that it is not clear what the syntax/form/phrasing of the language command is. Are the descriptions given in Fig. 5 examples of the language command or is the language command the list of objects? Does the syntax impact performance? Additionally, LACO outputs collision probability. While this is converted to a binary value (I believe through thresholding), it would be interesting to consider treating it as a cost in the context of an optimization-based motion planner. \n\nThe experiments evaluate LACO as compared to non-language conditioned planners and conduct several ablation studies. A variant of TrajOpt where LACO is used as the collision-checker is applied to a reaching and pushing task. For the pushing task, there are only three instances and the results are the same with and without conditioning on language, which does not support the claim that conditioning on language is useful in this task. Additionally, how is the downstream task success defined? For the failures, does this mean that there was an undesirable collision, no path was found or that the robot failed to reach the target? \n\nThe paper discusses several interesting limitations that serve as great areas of future work. In general, the paper is well-written and easy to follow. The related work seems sufficient. \n\nA few more minor questions and comments: \n- Does the single-view image include the robot? Sec 3.2 states that the environment configuration is agnostic to the robot joint states - is the robot masked out of the image? \n- Sec 4.2 mentions that state is respected to (x, y) position of the end effector. One interpretation (that seems unlikely) is that only the end effector is considered for collisions, not the entire configuration is the arm. Since I think this is the wrong conclusion, what does state mean in this case? Is it that, in collecting real world data, the objects and end effector position were only varied in (x, y)? \n- Since SceneCollisionNet is trained on a different data set, its unclear if the comparison is fair. However, I do not see this as a major concern, since SceneCollisionNet is not solving the same problem as LACO and the comparison is only one small part of the results. \n- For the simulation results, is the Built-In collision checker used as ground truth? \n- Sec 3.3, Line 134: \"implement LAPP using a sampling-based method, LAPP-TrajOpt [7].\" -> \"implement LAPP using an optimization-based method, LAPP-TrajOpt [7].\"\n- Sec 3.3, Line 135: \"additionally integrate a sampling base method, LAPP-RRT* [6]\" -> \"additionally integrate a sampling-based method, LAPP-RRT* [6]\"", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper suggests a new approach called "Language-Conditioned Path Planning" (LAPP) that relaxes the need for collision-free motion planning. LAPP works by allowing semantic information, like language commands, to determine which collisions are acceptable as it generates a path in the joint space. The paper presents an interesting, novel approach backed by extensive ablation studies and motion planning experiments.
Decision: Accept (Poster)