# Towards Open-Set NLP-Based Multi-Level Planning for Robotic Tasks
## 1 Introduction  
Allowing human operators to seamlessly interface with robotic control systems using unstructured natural language has been a long-held goal in artificial intelligence, quite possibly since the field's inception. Advances made in the field of machine learning over the past decade have opened the door to many exciting developments in this respect. Of great interest are transformer models for sequence prediction tasks [1], which may form the backbone of large language models (LLM) when applied in a pure natural language processing (NLP) context, or be combined with techniques for transforming multiple data modalities into the same latent embedding vector space to produce multi-modal--e.g., vision-language (VLM)--models [2].  
Traditionally, planning in robotics has been treated as a search problem in a configuration space [3]. This has advantages where the robot's environment can be understood analytically. However, when dealing with unstructured information such as a command given in natural language, explicitly expressing the task as a classic search problem becomes intractable. Recent work has leveraged large language models to act as high-level planners [4] and multi-modal vision-language-action models produce low-level control outputs directly [5]. The direct control approach is limited to acting on the information directly available to the trained model for inference, whereas the high-level planner can be more readily integrated with existing robot control software, utilize reusable workspace models with pre-computed constraints, and produce human-readable intermediate results.  
We seek to develop an architecture for a high-level planner-based robot control system which can start with a natural language prompt and produce a successfully executed sequence of motions producing the desired effect in the physical world. The goal of this article is therefore to demonstrate that it is possible to accomplish the above requirement through the implementation of the following three sub-systems:  
* A **high-level planning (HLP)** pipeline turning natural language commands into a structured plan and extract relevant queries for particular information from the raw text;
* A **low-level planning (LLP)** system capable of executing each action in the plan extracted from the command by the HLP;
* A **semantic map** of the environment that can be queried using natural language concepts obtained from the original command.  
Therefore, we outline an architecture for combining these subsystems into a robot control system. We implement an instance of this architecture as a proof-of-concept demonstrator, which can perform sequences of pick-and-place actions involving objects in a tabletop scene. We use an industrial robot arm as our actuator and a depth camera for perception. The demonstrator consists of an LLM-based HLP, an LLP system implementing a skill set of discrete actions, and either a static 2.5D semantic depth map or a fully 3D octree gridmap. We show successful end-to-end execution.  
In addition, quantization of the open-set image embeddings is assessed, as this provides a mitigation for the potentially prohibitive memory and network bandwidth costs of transferring large two-dimensional matrices of high-dimensional semantic embedding vectors, which were noted as part of demonstrator development. We demonstrate that in the admittedly favorable circumstances the system was tested under, embedding quantization produces very little observable variation in object retrieval.