# Towards Open-Set NLP-Based Multi-Level Planning for Robotic Tasks
## 4 Implementation
### High-Level Planning  
The main goal of high-level planning in this application is to transform an incoming user request, stated in natural language, into an action plan that the rest of the system can act upon. The plan is made through successive calls to the LLM that progressively populate prompt templates with information extracted from the user query. In this system, we use the following:  
* The LLama3 8B Instruct [32] model--an instruction fine-tuned LLM in the 8 billion parameter size range;
* The Q_4_K_M_ weights available at [33];
* Four-bit GGUF quantization implemented by _llama.cpp_[34]--to reduce the model's memory footprint. We use prompt chaining [35] to also reduce context length requirements. This also allows for additional programmatic post-processing of the output text between LLM calls. The HLP module processes inputs through the following stages: filtration, parsing, and planning. All stages utilize few-shot prompting [36], where each prompt contains examples for the task at hand. In filtration, incoming requests are classified into one of the predefined classes--_\"mobile_manipulation\"_,_\"query_answering\"_,_\"casual_discussion\"_ and_\"any_other_query\"_. This is used to trigger different execution paths in the following stages. Currently, only _\"mobile_manipulation\"_ is implemented, with other types of request terminating the high-level planning sequence. Filtration is achieved by invoking the model with the instruction prompt in Listing 1. ```
Youareanaturallanguageinterpreter. Youcanclassifyincomingrequestsunderoneofthepredefinedcategories. Thecategoriesthatyoucanclassifysentencesare: \"object_manipulation\",1-requestsconcerningfinding,moving,placingobjects; \"query_answering\",2-requeststhatneedknowledgetobeanswered; \"casual_discussion\",3-requeststhatareexpectingahumanresponse. \"Any_Other_Query\",0-forrequeststhatcannotbeplacedundertheother3
```  
Exampleoftheformatthatyouanswerin: Input:Whatalovelydayitis,isn'tit? Category:[\"casual_discussion\",3] ```
The parsing stage first identifies relevant objects and locations which would need to be found in the environment for the system to perform its task. It then breaks down the request into simpler sub-tasks (referred to as \"Steps\") for which planning can be performed separately, as in the Least-to-Most [37] approach. The object description consists of a placeholder, a textual description, and an object ID unique in the scope of the entire plan. An example of the JSON output format is given in Listing 2. ``` Input:Onthesmallblock,placethelargebox Output: {\"Query\":\"Onthesmallblock,placethelargebox\" \"Objects\":[[\"<small_block>\",\"smallblock\",0],[\"<large_box>\",\"largebox\",11], \"Steps\":[[\"Pickup<large_box>\",[1]],[\"Place<large_box>on<small_block>\",[0]]] } ```
Listing 2: JSON format for plan descriptors. The final stage is the actual planning--sequencing of actions. Each step created during \"parsing\" is processed by the planning stage as a list of actions from the available skill set, defined by the available action primitives. Chain-of-Thought (CoT) [38] prompting, which instructs the model to generate additional information before the final answer, is used to improve planning consistency and quality. To maintain continuity between steps, the expected state of the gripper(empty or holding an object) is generated as part of the output and passed as an input for the next step's planning operation. For example, the request in Listing 3 may produce an output such as the intermediate result in Listing 4. ```
Request:\"Place the<big_fish>onthe<small_plate>\"
```
Listing 3: Planning request. ```
Objects:[[\"<big_fish>\",\"bigfish\",3],[\"<small_plate>\",\"smallplate\",4]] Hand:[\"empty\"] Interpretation: MyHand:Free MyhandisFreeandIneedtopickupthe<big_fish>andplaceitonthe<small_plate>.Iwill:
1.Pick<big_fish>.myHand:<big_fish>
2.Placethe<big_fish>onthe<small_plate>.myHand:empty Nomoreactionsneeded.