# Voice Interaction Recognition Design in Real-Life Scenario Mobile Robot Applications
## 2 Human-Voice Interface System
### Automatic Speech Recognition (ASR)
#### 2.3.1 Conformer Model Architecture  
The conformer model is a modification of the encoder in the transformer framework, as it incorporates the conformer blocks of the CNN framework. The conformer architecture is shown in Figure 6. The left side is the classic transformer model architecture and the right side (red box) is the internal architecture after the modification. At the start, the spectrum of the audio signal output from the speaker separation system is inputted into the encoder; then, after spectrogram augmentation, it enters linearly in the convolution subsampling, dropout, and the main conformer blocks. Conformer blocks are executed N times according to the input size, just like the encoder in the transformer architecture.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Symbol** & **Meaning** \\ \hline \(X^{*}\) & Vector of output audio \\ \(\hat{X}\) & Vector of target audio \\ \(X_{T}\) & \(X^{*}\) project \(\hat{X}\) \\ \(X_{E}\) & \(X^{*}-\hat{X}\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Meaning of SDR symbol.  
Figure 5: SDR vector diagram.  
The internal structure of the conformer blocks is shown in Figure 7. It consists of three modules: feed-forward module, multi-head self-attention module, and convolution module, each of which uses a residual connection. The structure of the conformer blocks is like a sandwich, where the feed-forward module is located between the multi-head self-attention module and the convolution module, connecting them with each other; however, each feed-forward module only contributes half of its weight. Finally, it enters the post-layer norm for data processing, that is, the post-norm residual unit is utilized. Figure 8 [25] presents the Macaron-net [26] structure, which is formed through Equations (5)-(9). Conformer blocks are represented by x_i for the ith input, y_i for the output, FFN denotes the feed-forward module, MHSA denotes the multi-head self-attention module, and the convolution module is symbolized by Convol.  
\[\widetilde{x_{i}}=x_{i}+\frac{1}{2}FFN(x_{i}) \tag{5}\]  
\[x_{i}^{\prime}=\widetilde{x}_{i}+MHSA(\widetilde{x_{i}}) \tag{6}\]  
\[x_{i}^{\prime\prime}=x_{i}^{\prime}+Conv\big{(}x_{i}^{\prime}\big{)} \tag{7}\]  
\[\widetilde{x_{i}^{\prime\prime}}=x_{i}^{\prime\prime}+\frac{1}{2}FFN\big{(}x _{i}^{\prime\prime}\big{)} \tag{8}\]  
\[y_{i}=layernorm(\widetilde{x_{i}^{\prime\prime}}) \tag{9}\]  
Figure 6: Conformer model structure.  
Multi-head self-attention is the key structure in the transformer, and is a variation of self-attention. It is similar to an RNN, but it can input and output a whole set of data at the same time. Figure 9 shows the structure of the multi-head self-attention module, which uses the relative positional embedding and incorporates the pre-norm residual units with dropout (see Figure 10). Consequently, the convolution and feed-forward modules are also designed using the same structure. The transformer is unable to learn information on the sequence of the positions, so the developers of the transformer proposed to use position embedding for the multi-head self-attention to gain position information. Relative position coding provides better generalization over different input lengths and better robustness to variable speech lengths.  
Figure 8: Post-norm residual unit.  
Figure 7: Conformer blocks.