## 4 Experiments
### WSJ, AMI, and SWBD
#### 4.2.3 SA CTC/Attention  
In this section, the SA CTC/Attention architecture is conducted on the above three datasets. Based on the experiments and analyses in Section 4.2.2, we choose 5 as the \(attentionwindow\) size. In addition, the self-attention key, query, and value dimensions are all set 1024. To reduce computing complexity, we only experiment with single attention and fix the self-attention head quantity as 1. The results are shown in Table 3.  
For WSJ, although applying locate-aware attention with CTC contributes to improving the performance, the WER or CER remain constant in some cases. For example, in Table 3, the CER and WER of dev set have not been improved when decoding with RNNLM. However, the WER and CER of SA CTC/Attention achieve a consistent decline in all cases. In particular, it obtains 5.8% and 9.3% relative WER reduction for dev and test, when decoding with RNNLM. For AMI-IHM, LA CTC/Attention performs worse than the baseline as described in the previous section. However, compared with the baseline, the SA CTC/Attention achieves more than 3% performance improvement in all sets whether decoding with or without RNNLM. Particularly, it achieves 4.8% and 4.3% relative WER reduction for dev and eval with RNNLM. At the same time, it also acquires 4.8% and 3.3% relative WER reduction without RNNLM. For SWBD, the LA CTC/Attention already achieves overall performance improvement in all situations. Nevertheless, the SA CTC/Attention gets further improvement than the LA CTC/Attention. While decoding with RNNLM, the SA CTC/Attention acquires 8.6% and 7.1% relative WER reduction compared with the baseline. Furthermore, it achieves 8.4% and 7.0% relative WER decrease without RNNLM. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{
\begin{tabular}{c} **WSJ** \\ \end{tabular} } & \multicolumn{3}{c}{**LM Weight = 1**} & \multicolumn{3}{c}{**LM Weight = 0**} \\ \cline{2-10}  & & \multicolumn{2}{c}{**CER**} & \multicolumn{2}{c}{**WER**} & \multicolumn{2}{c}{**CER**} & \multicolumn{2}{c}{**WER**} \\ \cline{2-10}  & **dev** & **test** & **dev** & **test** & **dev** & **test** & **dev** & **test** \\ \hline Baseline & 3.9 & 2.4 & 8.6 & 5.4 & 7.5 & 5.7 & 22.2 & 17.7 \\ LA CTC/Attention & 3.9 & 2.4 & 8.6 & 5.1 & 7.5 & 5.5 & 22.0 & **16.7** \\ SA CTC/Attention & **3.7** & **2.2** & **8.1** & **4.9** & **7.4** & **5.5** & **21.5** & 16.8 \\ \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{3}{c}{**LM Weight = 1**} & \multicolumn{3}{c}{**LM Weight = 0**} \\ \cline{2-10}  & \multicolumn{2}{c}{**CER**} & \multicolumn{2}{c}{**WER**} & \multicolumn{2}{c}{**CER**} & \multicolumn{2}{c}{**WER**} \\ \cline{2-10}  & **dev** & **test** & **dev** & **test** & **dev** & **test** & **dev** & **test** \\ \hline Baseline & 22.3 & 23.2 & 35.1 & 37.4 & 23.4 & 24.7 & 39.6 & 42.2 \\ LA CTC/Attention & 22.3 & 23.6 & 35.3 & 38.1 & 23.5 & 25.1 & 40.1 & 43.1 \\ SA CTC/Attention & **21.0** & **22.1** & **33.4** & **35.8** & **22.1** & **23.7** & **37.7** & **40.8** \\ \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{3}{c}{**LM Weight = 1**} & \multicolumn{3}{c}{**LM Weight = 0**} \\ \cline{2-10}  & \multicolumn{2}{c}{**CER**} & \multicolumn{2}{c}{**WER**} & \multicolumn{2}{c}{**CER**} & \multicolumn{2}{c}{**WER**} \\ \cline{2-10}  & **eval** & **rt03** & **eval** & **rt03** & **eval** & **rt03** & **eval** & **rt03** \\ \hline Baseline & 30.2 & 31.9 & 47.8 & 50.6 & 30.8 & 32.5 & 51.0 & 54.0 \\ LA CTC/Attention & 29.4 & 31.5 & 47.3 & 50.3 & 30.1 & 32.4 & 50.5 & 53.8 \\ SA CTC/Attention & **26.9** & **29.0** & **43.7** & **47.0** & **27.8** & **30.0** & **46.7** & **50.2** \\ \hline \hline \end{tabular}
\end{table} 
Table 3: The performances of the SA CTC/Attention for WSJ, AMI, and SWBD compared with the LA CTC/Attention and baseline architectures. Apart from the CER and WER improvement, self-attention is very helpful in accelerating the convergence during training.