# Towards Open-Set NLP-Based Multi-Level Planning for Robotic Tasks
## 5 Experimental Setup and Results
### HLP Performance  
Position the energy drink bottle over the toy car, then rest the pair of scissors on the multimeter. While the above tests give some indication of semantic map performance, they give no insight into the ability of the HLP to reliably decompose commands into action sequences. One way to verify such a system is to start with known plans, construct commands from them and test whether the HLP is able to recover the original plan. As the current system is limited to a very simple set of actions, the following three types of command were selected to test for different potential failure modes: \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & \multicolumn{6}{c}{**Recall \%**} \\ \cline{2-7}
**Query Text** & \multicolumn{3}{c}{**Static Depth Map**} & \multicolumn{3}{c}{**VectorOcTree Map**} & \multicolumn{1}{c}{
\begin{tabular}{c} **Seen \(n\)** \\ Times** \\ \end{tabular} } \\ \cline{2-7}  & **Ternary** & **Byte** & **Float32** & **Ternary** & **Byte** & **Float32** \\ \hline protective eyewear & 75 & 75 & 75 & 75 & 100 & 75 & 4 \\ pink foam & 100 & 66.67 & 66.67 & 100 & 100 & 100 & 3 \\ grey key & 100 & 100 & 100 & 100 & 100 & 100 & 1 \\ realsense box & 100 & 100 & 100 & 100 & 100 & 100 & 2 \\ wrench & 100 & 100 & 100 & 100 & 40 & 100 & 5 \\ computer mouse & 100 & 100 & 100 & 100 & 100 & 100 & 5 \\ white bottle & 40 & 60 & 60 & 60 & 60 & 60 & 5 \\ multimeter & 100 & 100 & 100 & 100 & 66.67 & 66.67 & 3 \\ blue stapler & 100 & 100 & 100 & 100 & 100 & 100 & 3 \\ toy car & 100 & 100 & 100 & 80 & 60 & 100 & 5 \\ red lego & 100 & 100 & 100 & 100 & 100 & 100 & 3 \\ scotch tape & 100 & 100 & 100 & 100 & 100 & 100 & 5 \\ pair of scissors & 100 & 100 & 100 & 100 & 100 & 100 & 4 \\ energy drink bottle & 66.67 & 100 & 100 & 100 & 100 & 3 \\ tape measure & 75 & 75 & 75 & 100 & 75 & 100 & 4 \\ blue lego & 100 & 100 & 100 & 100 & 100 & 100 & 1 \\ \hline mean recall & 91.04 & 92.29 & 92.29 & 94.69 & 87.6 & 93.85 & \(\Sigma=56\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Object queries and recall rates according to map type and vector quantization level. * \"simple\"--simple verb-object statements, corresponding to a single \"pick\" \(\rightarrow\) \"place\" action sequence, to test for robustness in wording;
* \"forward\"--composite statements of multiple (in this case, two) simple pick-place sequences, to test for the model's ability to correctly isolate actions from each other;
* \"reverse\"--composite statements reversed sentence order, e.g., \"do B, but only after doing A\", to test the model's ability to recover correct sequencing. Due to the absence of an existing data set, a method for collecting such commands is necessary. LLMs, by construction, approximate the distribution of natural language sequences quite well. However, using the same model as in the HLP would result in test sequences reflecting the same biases. To minimize the chances of encountering such effects, it was decided to sample statements using a different large language model. In this case, the interactive assistant API of _OpenAI_'s flagship _GPT-4o_[49] was used. Specifically, generating the test sequences was accomplished by following the prompt chain in Listing 5, where the previous prompt and the resulting model output are included in the context of the subsequent model invocation along with the new prompt. To further obfuscate the required actions, we also take an additional step to make each command resemble more typical human conversation by continuing the process with the prompts in Listing 6. ```
Step1:Generate20 differentwaystosay\"pickupobjectAandputitdownonobjectB\".Puteachvariationinarewellineprecededbyitsindex.Usecreativewordingbutkeepthemeaningthesame.... Step2:Produceashortenedversionofeach,e.g. \"pickupobjectAandputitdownonobjectB\"->\"putobjectAonobjectB\"... Step3:Foreachpairi,i+1,writeawaytcombinethestatements: \"PutobjectAonobjectB.Then,setobjectConobjectD.\" finishbyloopingbackaround.... Step4:createversionofeachlinewhereCandDarementionedbeforeAandB, buttheactionwithAandBisstilltobeperformedfirst,e.g.