# Towards Open-Set NLP-Based Multi-Level Planning for Robotic Tasks
## 5 Experimental Setup and Results
### HLP Performance
\begin{cases}\text{planning, if }\epsilon_{p}\\ \text{structure, if }\
eg\epsilon_{p}\wedge\epsilon_{s}\\ \text{object, if }\
eg\epsilon_{p}\wedge\
eg\epsilon_{s}\wedge\epsilon_{o} \end{cases}  \tag{12}\]

This means that the total used to compute the percentage in the \"object\" column of the table only counts instances without planning or structural errors, and \"structure\"--onlythe instances without planning errors. We collect the \(\tau,p_{i}(\tau)\) sequence, its corresponding text queries and error flags into files, which have been included in the _hlp_results.zip_ archive of the reference data set cited in the data availability statement at the end of this article. Also included are files in which only the lines corresponding to a specific error type and query type are collected. The results of this comparison are collected in Table 2. The rows correspond to command structure, the two major column blocks--to whether the dry or extended version of the command is being tested. Each column then records the frequency of a particular error type given a command type. The rightmost row records the bottom line--the percentage of correctly recovered plans across both styles of language. The bottom row combines the results across all command types. ## 6 Discussion

Given a sufficiently constrained scenario--pick-and-place operations on a flat surface within the robot's field of view and reach--it is clear, as evidenced by the successful end-to-end execution in Figure 5, that it is indeed possible to implement a natural language-based robot control system following the approach outlined in this article, which answers the main question outlined by the authors at the start of this article. Examining the results of the HLP performance evaluation, it is clear that, at least for the three simple types of plans presented in the test data set, adequate performance is attainable with the methods used. However, as is seen in Table 2, a counter-intuitive result can be observed--in that the error rates in the forward version of the composite query are higher than those in the reverse. Closer examination of the corresponding files in the reference data set seems to show a pattern where additional actions are hallucinated as in Listing 8 or objects are duplicated as in Listing 9. ```
#hlp_results/dry/eval_forward_err_struct.txt-1 #query Set theggles on the grey key, then position the toy car over the white bottle. #plan ['pick', 'place', 'pick', 'place', 'place'] #objects ['ggggles', 'grey key', 'toy car', 'white bottle', 'toy car'] \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multirow{2}{*}{\
\begin{tabular}{c} **Command** \\ **Type** \\ \end{tabular} } & \multicolumn{6}{c}{**Dry**} & \multicolumn{6}{c}{**Extended**} & \multirow{2}{*}{
\begin{tabular}{c} **Correct** \\ **\%** \\ \end{tabular} } \\ \cline{2-2} \cline{5-10}  & \multicolumn{6}{c}{**Error \% by Type (of Total)**} & & & & & **\%** \\ \cline{2-2} \cline{5-10}  & **Planning** & **Structure** & **Object** & **Any** & **Planning** & **Structure** & **Object** & **Any** & \\ \hline simple & 00.00 & 0.51 & 00.00 & 0.51 & 0.00 & 0.51 & 0.00 & 0.51 & 99.49 \\ \hline forward & 1.50 & 1.50 & 1.50 & 4.50 & 3.50 & 6.00 & 1.00 & 10.50 & 92.50 \\ \hline reverse & 1.00 & 0.50 & 0.50 & 2.00 & 2.00 & 1.00 & 0.50 & 3.50 & 97.25 \\ \hline all & 0.84 & 0.84 & 0.67 & 2.35 & 1.84 & 2.51 & 0.50 & 4.86 & 96.40 \\ \hline \hline \multicolumn{10}{c}{Data set size} \\ \hline Command & \multirow{2}{*}{Dry} & \multirow{2}{*}{Extended} & \multirow{2}{*}{Total} \\ type & & & & & & & & \\ \hline simple & & 197 & & & 197 & & 394 \\ \hline forward & & 200 & & & 200 & & 400 \\ \hline reverse & & 200 & & & 200 & & 400 \\ \hline all & & 597 & & 597 & & 1194 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: HLP performance by command type.