## 4 Semantic Understanding-Based Task Slot Structure through Visual and Auditory Channels
### Extracting Semantic Texts for Hand Posture- and Speech-based Commands Using Semantic Understanding
#### 4.1.2 Sequence Tagging of Instructions  
To solve the first problem, linear chain Conditional Random Fields are used to tag words in instructions. Furthermore, the words of these instructions are classified by analyzing the characteristics of the commands, which is shown in Table 5. To tag \"dis_g\", \"dis_s\" and \"obj\" classes (Which is shown in Table 5) correctly, a pre-established lexicon and Amap API (the application programming interface provided by AutoNavi) are used to identify their types. Because the motion of the integrated leg-arm hexapod robot is complex, speech instructions in human-robot interaction are more complex than ordinary mobile robots, and key information cannot be extracted from instructions directly. Thus, the chunk analysis method is applied to analyze the instructions, and we find that the effective information, which is not affected by syntactic structure, can be divided into several chunks: motion type \"V\", motion direction \"DIR\", stop position \"DIS\", motion speed \"VE\", moving gait \"BT\" and operation objects. Moreover, the operation objects are divided into two categories: body of the robot \"USE\" and external object \"OBJ\". Additionally, instructions are tagged by the chunks above, and classes compose chunks. To elaborate chunks clearly, a result of chunk analysis of an instruction (Put the dangerous goods, which is in the a305 room, into the explosion-proof tank.) was given, which is shown in Figure 4. As shown in Figure 4, a Chinese instruction is on the top row of the table, the classes of Chinese words are listed on the bottom row of the table. For the sake of an English reader's convenience, Chinese words were translated from Chinese into English, which are listed below the corresponding Chinese words. The words and classes in the blue bounding box belong to external object chunk \"OBJ\", while the words and classes in the green bounding box belong to stop position chunk \"DIS\". \begin{table}
\begin{tabular}{c c} \hline
**Speech Instruction** & **Results of the Word Segmentation and POS Tagging** \\ \hline Put the dangerous goods, which is in the a306 room in the New Main building, into the explosion-proof tank. & with/pba New/a (Main building)/n a306/n room/n in/f of/ude1 (dangerous goods)/n put/v into/v proof/v \\ Put the speculum into the crevice of the door slowly. & explosion/v (into the tank)/s \\ \hline \end{tabular}
\end{table} 
Table 4: Results of the word segmentation and POS (part-of-speech) tagging of Chinese instructions. Because an instruction often has multiple clauses, if punctuation is used as the basis for segmentation (speech is recognized as text, so, there is punctuation in the text), the problems are that a clause may imply multiple actions and that two verbs in different clauses form one action. Some examples are shown in 34. To solve this problem, the instruction is segmented into clauses based on the assumption that only one action is conveyed in a clause. * Go to room a306 to get inflammable substances. * Move forward until you reach the Weishi Hotel. After three steps of tagging, instructions are decomposed into clauses, the clauses are composed of chunks, and chunks are comprised of classes. A diagram of structure of instructions is shown in Figure 5. It can be seen that each word in an instruction can be described accurately. Conditional Random Fields (CRFs) is a kind of undirected graphical model [20], in which context features are considered, and all features are normalized to obtain a global optimal solution. Thus, CRFs suit to label sequence data. Moreover, the size of our corpus is relatively small, for which it is suitable to use a supervised learning algorithm such as CRFs. Thus, CRFs were selected in this work. Figure 4: Result of chunk analysis of an instruction.