# PDDL Planning with Natural Language-Based Scene Understanding for UAV-UGV Cooperation
## 4 Experiment
### Results  
The experiment involving patrolling and finding a missing child was successful. In this study, we used 16 POIs for robot patrolling according to the assigned area. When the child went missing, assume that a human is present at POI 9. Then, the robots were asked to check all the POIs and find a human who is likely to be the missing child. When such a human is detected, a scene graph is generated and sent to the DICQ.R. Using the achieved semantic information, a POI is added and the closest robot is asked to go to POI to check if the detected human is the missing child. Tables 1 and 2 show the generated plans for the robots. In the initial plan, POI 16 is not included. After the human is detected by Beyond, a new POI (16) is generated and is checked by Smart cookie.  
We used the XML file structure to send the semantic graphs to DICQ.R. The XML/OWL parser located inside the knowledge engine is provided triplet data that contain scene graph information. The OWL file is generated by the classification processes of object property and data property relations. The object property relation is relevant to the relationship between objects, and the data property relation is relevant to the properties of these objects. According to the command from the DICQ.R, which provides the mission parameters, JENA-TDB fetches semantic information using SPARQL and sends it to the environment modeler. For example, using the received triple data of \"human-behind-tree,\" \"behind\" is saved as \"owl:ObjectProperty rdf:about plan:behind/.\" \"human-hasPositionX-100\" is saved as \"owl:DataProperty rdf:about plan:hasPositionX.\" Also, objects are parsed as \"owl:NamedIndividual,\" which is used to describe instances.  
Figure 7: Overall scenario outline: Using one UAV and three AGVs, patrolling was conducted and the missing child was found.  
The generated language descriptions and scene graphs are shown in Figure 8a,b. The language descriptions and scene graphs were successfully generated in the simulation environment. As illustrated in Figure 8c, we utilized the natural language-based scene understanding across two situations: (1) language description in the \"failed mission situation\" to inform the control tower about the current situation, and (2) the scene graph in the \"human detected situation,\" to add a POI to verify whether the detected human is the missing child. As a result, we verified that the proposed framework could successfully perform the required planning using heterogeneous multiple robots based on natural language-based scene understanding.  
\begin{table}
\begin{tabular}{l l l} \hline \hline
0.000: & (goto\_point\_indoor cookie0 POI0 POI0) & [20.000] \\
0.000: & (goto\_point\_outdoor cookie1 POI12 POI12) & [20.000] \\
0.000: & (goto\_point\_street cookie2 POI2 POI2) & [20.000] \\
0.000: & (fly beyond0 POI6 POI6) & [20.000] \\
20.001: & (goto\_point\_indoor cookie0 POI0 POI1) & [20.000] \\
20.001: & (goto\_point\_outdoor cookie1 POI12 POI13) & [20.000] \\
20.001: & (goto\_point\_street cookie2 POI2 POI3) & [20.000] \\
20.001: & (fly beyond0 POI6 POI7) & [20.000] \\
40.001: & (goto\_point\_indoor cookie0 POI1 POI10) & [20.000] \\
40.001: & (goto\_point\_outdoor cookie1 POI13 POI14) & [20.000] \\
40.001: & (goto\_point\_street cookie2 POI3 POI4) & [20.000] \\
40.001: & (fly beyond0 POI7 POI8) & [20.000] \\
60.001: & (goto\_point\_indoor cookie0 POI10 POI11) & [20.000] \\
60.001: & (goto\_point\_outdoor cookie1 POI14 POI15) & [20.000] \\
60.001: & (goto\_point\_street cookie2 POI4 POI5) & [20.000] \\
60.001: & (fly beyond0 POI8 POI9) & [20.000] \\
80.001: & (detect beyond0 POI9 human) & [20.000] \\ \hline \hline \end{tabular}
\end{table}
Table 1: Generated plan for Part 1 of the scenario.  
\begin{table}
\begin{tabular}{l l l} \hline \hline
0.000: & (goto\_point\_outdoor cookie1 POI15 POI16) & [20.000] \\ \hline \hline \end{tabular}
\end{table}
Table 2: Generated plan for Part 2 of the scenario.  
Figure 8: _Cont._