# PDDL Planning with Natural Language-Based Scene Understanding for UAV-UGV Cooperation
## 3 Architecture
### Natural Language-Based Cognition  
Cognition part in scene understanding engine is largely composed of semantic graph generation, language description, and scene graph generation as shown in Figure 3. To understand the surrounding environment in natural language, we generate a natural language description and scene graph. In this study, we assume that the robots use a graph map (for motion planning) generated using semantic SLAM, which is a widely used environment representation method in robotics [7]. To utilize the graph map \(\mathcal{G}=(\mathcal{V},\mathcal{E})\) that contains features and positions of the detected object as nodes \(v_{i}\in\mathcal{V}\) and their relationships as edges \(e_{ij}=(v_{i},v_{j})\in\mathcal{E}_{ij}\), we closely follow [PERSON] and [PERSON] [41] for generating the language description and graph inference phase of [PERSON] et al. [42] for the scene graph generation. However, since the edge information of the graph map is binary, which can only infer whether a connection exists or not, or a weighted value that indicates relations such as the Euclidean distance between objects, it is difficult to find the semantic meaning. Therefore, we additionally extract features of the union region of two objects for edge information. For each \(v_{i}\) and \(e_{ij}\), features are extracted using VGGNet [43]. \(f_{i}^{v}\) is the feature vector of \(v_{i}\), and \(f_{ij}^{ce}\) is the feature vector of \(e_{ij}\). \(p_{i}\) is position vector of \(v_{i}\).  
Figure 3: On language description part, a GCN extracts features from the graph map. The extracted graph feature is concatenated with a word and feed into the RNN as input. Then, the RNN generates sentence attention over the graph. On scene graph generation part, Two different message pooling methods are performed. Node message pooling uses the inbound and outbound edge states with a node. Edge message pooling uses the object states with an edge. This process is repeated to precisely predict the natural language words corresponding to the nodes and edges of the graph.  
Figure 2: General overview of the proposed architecture.  
A GCN with graph convolution layers defined by spectral graph theory and fully connected layers is utilized to extract features from irregular and non-Euclidean graphs. Then, an RNN is used to generate a language description over the graph. The RNN takes the encoded graph features concatenated with a word vector and predicts the probabilistic distribution of the target word vector. Given that we also back-propagate the GCN when training the RNN, we can expect that graph features that fit the generated sentence will be extracted. The generated description can be used to understand the surrounding environment when an unexpected situation occurs.  
Scene graph generation involves the process of finding appropriate words corresponding to each node and edge of the graph. We denote variables that need to be predicted as \(\mathbf{g}=(v_{i}^{class},e_{ij}\mid i=1\dots n,j=1\dots n,i\
eq j)\), where \(\mathcal{C}\) is a set of object classes and \(\mathcal{R}\) is a set of relationship types, \(v_{i}^{class}\in\mathcal{C},e_{ij}\in\mathcal{R}\). The optimal \(g^{*}\) is found as follows:  
\[\mathbf{g}^{*}=argmax_{\mathbf{g}}Pr(\mathbf{g}\mid f_{i}^{v},f _{ij}^{e}) \tag{1}\] \[Pr(\mathbf{g}\mid f_{i}^{v},f_{ij}^{e})=\prod_{i\in V}\prod_{j \
eq i}Pr(v_{i}^{class},e_{ij}\mid f_{i}^{v},f_{ij}^{e}) \tag{2}\]  
The iterative message pooling method based on the gated recurrent unit (GRU) is utilized. Edge features and node features are fed into the edge GRU and node GRU as the initial value, respectively. After the message pooling, the edge message is fed into the edge GRU and the node message is fed into the node GRU. The iteration that follows precisely predicts words for the nodes and edges. The scene graph can be used to gather environmental information in natural language for large and unstructured environments.