# Grammar-Supervised End-to-End Speech Recognition with Part-of-Speech Tagging and Dependency Parsing
## 5 Conclusions  
In this study, we have proposed a method for incorporating grammar knowledge into automatic speech recognition systems. We have shown that explicit grammar constraints are an effective way to improve the recognition performance of ASR systems and increase the readability of the speech hypotheses. Specifically, we have proposed several grammar constraint methods, including the reinforcement of word embedding with grammar embedding and adding the auxiliary text-to-grammar downstream task, which can intensify the grammar rules of the final recognition hypothesis. For providing various evaluation methods for grammar knowledge in recognition outputs, we introduced an evaluation method for the grammatical deviation distance and a grammar-based mean opinion score to supplement the existing common evaluation criteria such as WER and CER for exploring an expandable usage paradigm with grammar knowledge. To evaluate the effectiveness of our approach, we conducted experiments on the small open-source Mandarin speech corpus AISHELL-1 and large private-source Mandarin speech corpus TRANS-M to verify the effectiveness of our method. Compared with the baseline, our approach achieved relative CER reductions of 3.2% and 5.0% and relative GDD reductions of 4.7% and 5.9% on AISHELL-1 and TRANS-M tasks, respectively. Meanwhile, the grammar-based mean opinion score of our method is about 4.29 and 3.20, while the baseline can only reach 4.11 and 3.02. Overall, our study provides a framework for future research on incorporating grammar knowledge into ASR systems. We believe that this approach has great potential for improving speech recognition technology and enhancing the user experience in various applications such as virtual assistants, voice-controlled devices, and automated customer service systems. Future work will evaluate the performance of deeper integration with the grammar information, and we will also incorporate more downstream tasks to increase the role of multiple evaluation methods. \begin{table}
\begin{tabular}{c c c} \hline \hline
**ID** & **Model** & **Utterance** \\ \hline \multirow{4}{*}{1} & Ground-Truth & Chinese: \(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\) \\  & & English: You know what love is \\  & Baseline & Chinese: \(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\) \\  & & English: You know what it is \\  & MEMG & Chinese: \(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\) \\  & & English: You know what love is \\  & Ground-Truth & English: Implement full caliber statistics and dynamic detection \\
2 & Baseline & Chinese: \(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\) \\  & & English: Implement net power statistics and dynamic detection \\  & MEMG & Chinese: \(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\)\(\mathbb{F}\) \\  & & English: Implement full caliber statistics and dynamic detection \\ \hline \hline \end{tabular}
\end{table} 
Table 7: The examples of two test sets. **Author Contributions:** Conceptualization, G.W. ; methodology, G.W. and J.Z. ; project administration, J.G. ; software, T.M. ; supervision, J.G. and Z.Y. ; validation, G.W. and T.M. ; writing\(-\)original draft, T.M. ; writing\(-\)review & editing, G.W., J.Z. and H.C. All authors have read and agreed to the published version of the manuscript. **Funding:** The work was supported by the National Key R & D Program of China (Open Innovation Platform on Intelligent Voice, 2020 AAA0103600). **Institutional Review Board Statement:** Not applicable. **Informed Consent Statement:** Not applicable.