# Towards Open-Set NLP-Based Multi-Level Planning for Robotic Tasks
## 4 Implementation
### Low-Level Planning  
Low-level planning (LLP) is responsible for planning and controlling the execution of each step in the plan provided by the HLP subsystem on physical hardware. It is tightly coupled to the low-level control aspects of the robot system and therefore developed within the _ROS 2_ framework for inter-process communication and task scheduling, with the corresponding version of _Movelt_[39] employed for motion planning and control.  
A set of five action primitives is defined following the action-activity hierarchy defined in [40]: **embed**, **find**, **get_grasp**, **move** and **activate_gripper**. These action primitives can be divided into subgroups such as for perception, consisting of **embed**, **find**, **get_grasp** and for robot motion, which includes **move** and **activate_gripper**.  
The **perception primitives** are defined as ROS 2 service calls that wrap REST API calls for communicating with external components. The client side is directly embedded in the LLP software module as a function or method call, while the server node handles the REST API calls. The perception primitives may be viewed as functions:  
* **embed**: string \(\mapsto\mathbb{R}^{\textit{semantic\_dim}}\)--wrapsa call to the VLM text embedder API, returns the latent space embedding of the input text;
* **find**: \(\mathbb{R}^{\textit{semantic\_dim}}\mapsto\textit{Option}[\mathbb{R}^{3}]\)--wrapsa map API call with an embedded object descriptor string, and converts the camera-frame position \(\mathfrak{t}^{\textit{C}}\) to base-frame position \(\mathfrak{t}^{\textit{B}}\) (may return empty when no object found);
* **get_grasp**: \(\mathbb{R}^{\textit{semantic\_dim}}\mapsto\textit{Option}[\text{SE3}]\)--returnsthebase-frame grasppose \(T^{\textit{B}}_{\textit{grasp}}\) for an object described by the embedding vector (may return empty when no object found).  
The **robot motion primitives** abstract robot operation. Currently only two of these are implemented--sufficient for simple pick-and-place operations required by the minimal proof-of-concept architecture:  
* **move**: SE3 \(\mapsto\) bool--wraps planning and execution calls through _MoveIt_, potentially using a mix of joint-space and Cartesian paths depending on the goal and current state of the robot, returns a flag indicating whether the trajectory was completed successfully;
* **activate_gripper** : bool \(\mapsto\) bool--wraps calls to a ROS action server that open or close the gripper, returns a flag indicating whether the target gripper state was achieved.  
The primitives are treated as composable subroutines to form **actions**. Two actions have been implemented in this system--**pick** and **place**. Each is defined as a procedure that invokes the primitives as subroutines, depicted in Figure 3. These actions define the skill set of the system, and plans created by the HLP are expressed as sequences of these actions.  
A communication interface consistent with all actions implemented by the LLP system is implemented in the form of two ROS message types:  
* **ActionDesc**--the argument for each action call
* _sequence id_ : int
* _action type_ : string
* _objects_ : list[ObjDesc]  
* _text description_ : string
* _latent description_ : Option[\(\mathbb{R}^{semantic\_dim}\)]
* _object position_ : Option[\(\mathbb{R}^{3}\)]
* _grasp pose_ : Option[SE3]  
Fields of an optional type are populated by the perception primitives if and when required.  
The LLP module is directly integrated with the main execution loop and user interface of the system. Therefore, HLP requests are made by the LLP code through a middleman _ROS_ service which abstracts the external HLP interface. This returns a sequence of **ActionDesc** objects. These are iterated over in sequence order, and the action type string is used to select the corresponding action to invoke with each.