# Grammar-Supervised End-to-End Speech Recognition with Part-of-Speech Tagging and Dependency Parsing
## 4 Experiments
### Experiment Results
#### 4.4.3 Multiple Evaluation Methodology of Grammar (MEMG)  
To make the best use of the grammar information, we combine both the part-of-speech loss and the dependency parsing loss into the original loss simultaneously. As described in Table 3, the results on both AISHELL-1 and TRANS-M show that the multiple evaluation methodology of grammar obtains a very significant performance improvement in terms of the CER and GDD. Specifically, the AISHELL-1 of the CER decreased from 4.87% to 4.71% for the test set that has a 3.29% relative reduction, while TRANS-M of the CER decreased from 18.14% to 17.23% with a 5.0% relative reduction. Meanwhile, compared with the baseline model, the relative GDD reduction of AISHELL-1 and TRANS-M can reach 4.7% and 5.9%, respectively, which indicates the improvement of readability of the final speech recognition hypothesis.  
In order to know the influence of the different evaluation methodologies on the ASR system, we designed a series of ablation experiments with different hyper-parameters to balance the tasks. As shown in Table 4, different grammatical weights are set to make certain the constraining function of the different evaluation methodologies.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Evaluation Methodology**} & \multirow{2}{*}{**Weight**} & \multicolumn{4}{c}{**AISHELL-1 (\%)**} \\ \cline{3-6}  & & \multicolumn{2}{c}{**Dev**} & \multicolumn{2}{c}{**Test**} \\ \cline{3-6}  & & **CER** & **GDD** & **CER** & **GDD** \\ \hline \multirow{5}{*}{DP} & 0 & 4.42 & 5.48 & 4.87 & 6.16 \\  & 0.1 & 4.50 & 5.57 & 4.88 & 6.03 \\  & 0.3 & 4.42 & 5.43 & 4.86 & 6.06 \\  & 0.5 & 4.40 & 5.43 & 4.82 & 6.02 \\  & 0.7 & 4.45 & 5.47 & 4.93 & 6.17 \\ \hline \multirow{5}{*}{POS} & 0 & 4.42 & 5.48 & 4.87 & 6.16 \\  & 0.1 & 4.31 & 5.26 & 4.78 & 5.84 \\ \cline{1-1}  & 0.3 & 4.31 & 5.16 & 4.72 & 5.84 \\ \cline{1-1}  & 0.5 & 4.34 & 5.26 & 4.69 & 5.80 \\ \cline{1-1}  & 0.7 & 4.27 & 5.17 & 4.80 & 5.83 \\ \hline \hline \end{tabular}
\end{table}
Table 4: The ablation experiments of MEMG.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{2}{c}{**AISHELL-1 (\%)**} & \multicolumn{2}{c}{**TRANS-M (\%)**} \\ \cline{2-5}  & **CER** & **GDD** & **CER** & **GDD** \\ \hline Baseline & 4.87 & 6.16 & 18.14 & 24.02 \\ RWE (ground truth) & 4.49 & 5.50 & 15.30 & 21.02 \\ RWE (hypothesis from & 4.84 & 6.16 & 18.03 & 24.00 \\ CTC branch) & 4.83 & 6.08 & 17.73 & 23.06 \\ AT2G & 4.71 & 5.87 & 17.23 & 22.60 \\ \hline \hline \end{tabular}
\end{table}
Table 3: The CER and GDD of the proposed system on two test sets.  
Compared with the DP task, the POS task is more effective for the ASR system within the grammar constraint. It can be explained that learning the structure of a whole utterance is challenging and usually requires adequate extended information. Meanwhile, we also note that the lower CER does not necessarily mean a lower GDD, which is almost consistent with our guess and observation. When the weight is 0.5, we find the model has almost achieved consistency improvement.