## 4 Semantic Understanding-Based Task Slot Structure through Visual and Auditory Channels
### Extracting Semantic Texts for Hand Posture- and Speech-based Commands Using Semantic Understanding
#### 4.1.2 Sequence Tagging of Instructions
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Classes** & **Example** & **Description** & **Classes** & **Example** & **Description** \\ \hline v1 & Move & Verbs that should be executed in order & pre & to & Preposition \\ v2 & speed up & Verbs that should be executed immediately & dir & front & Direction \\ obj & dangerous & Entity can be recognized & & & \\  & goods & through the image & t & 2 min & Time \\  & & recognition and pronoun & & & \\ dis & 5 m & Distance & tb & 4 + 2 & Gait \\ dis\_g & Vision Hotel & Spots on the outdoor map & ve & fast & Speed \\ dis\_s & elevator room & Spots on the indoor map & wj &, & Punctuation \\ use & right hand & Robot body & o & at & Others \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Classes of words. Figure 5: Diagram of structure of instructions. Specifically, linear CRFs were used, in which \(y=(y_{1},y_{2},\ldots,y_{n})\) denotes the probability of a label sequence, and \(x=(x_{1},x_{2},\ldots,x_{n})\) represents an observation sequence, which is shown as follows:  
\[ \begin{split} P(y\big{|}x)&=\frac{1}{Z(x)}\exp\left( \sum_{i,k}\lambda_{k}t_{k}\Big{(}y_{i-1},y_{i},x,i\Big{)}+\sum_{i,l}\mu_{l}s_{l }\Big{(}y_{i},x,i\Big{)}\right)\\ Z(x)&=\sum_{y}\exp\left(\sum_{i,k}\lambda_{k}t_{k} \Big{(}y_{i-1},y_{i},x,i\Big{)}+\sum_{i,l}\mu_{l}s_{l}\Big{(}y_{i},x,i\Big{)} \right)\end{split}  \tag{1}\]  
where \(t_{k}\) is a transition feature function, \(s_{l}\) is a state feature function, \(\lambda_{k}\) and \(\mu_{l}\) are weights, and \(Z(x)\) is the normalization factor. Three steps of instructions tagging both use CRFs for sequence tagging, and the three steps constitute the Cascaded Conditional Random Fields (CCRFs), as shown in Figure 6. The input variables of the upper layer contain not only the observation sequences, but also the recognition results of the lower layers, which increases the types of features of upper layers, and it is helpful to complete the complex semantic sequence tagging. To solve the second problem, letter-based features instead of word-based features are adopted to correct the position of segmentation of the words. To get the position of the letter in word and class, we utilize \"BMEWO\" to mark the position. Specifically, in \"BMEWO\", \"B\" denotes the letter that is in the first place of the word or class, \"M\" represents the letter that is in the middle of the word or class, \"E\" means the letter that is in the end of the word or class, \"W\" denotes the single letter that constitutes the word or class, \"O\" means the letter that does not belong to any class. Features used for each layer of CRFs are shown in Table 6, in which W is a letter, P is the comprehensive feature of the position of the letter in the word and the POS tagging (like \"B_v\"); T is the result of the layer 1 and U is the result of the layer 2; W(0) is the current letter, W(1) is the next letter, W(2) is the second letter after W(0), W(-1) is the letter preceding W(0) and W(-2) is the second letter before W(0). The definition of P(n), T(n) and U(n) are the same to W(n).