## 5 Conclusions  
In order to improve the performance of hybrid CTC/Attention end-to-end ASR, this paper proposes integrating attention mechanism with the CTC branch to address CTC's output independent assumption. Firstly, we explore time-restricted location-aware attention CTC/Attention, establishing the proper time-restricted attention window size. \"Time-restricted\" indicates that the attention mechanism is conducted on a limited window of frames to the left and right. Inspired by the success of self-attention in machine translation, we further introduce the time-restricted self-attention CTC/Attention that can better model the long-range dependencies among the frames. For a fair comparison, our experiments are carried out on three clean datasets, WSJ, AMI-IHM, and SWBD. For WSJ, the SA CTC/Attention obtains 5.8% and 9.3% relative WER reduction in dev and test with RNNLM. For AMI-IHM, it achieves 4.8% and 4.3% relative WER decrease in dev and eval with RNNLM. For SWBD, it reaches 8.6% and 7.1% relative WER reduction in eval and rt03 with RNNLM. With the exception of WER and CER evaluation criteria, we also exhibit the details of the training process and present the loss curves. The lowest loss curves of SA CTC/Attention also verify the effectiveness of the algorithm. To explore the robustness of this method to noise and reverberation, we also experiment with the SA CTC/Attention framework in CHIME-4. The results indicate that it achieves 11.2% and 7.7% relative WER reduction in et05 simu and et05 real.  
Figure 4: The train and valid loss of the baseline, LA CTC/Attention and SA CTC/Attention architectures for CHIME-4.  
\begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**CHIME-4**} & \multicolumn{5}{c}{**et05 simu**} & \multicolumn{5}{c}{**dt05 simu**} \\ \cline{2-11}  & & **PED** & **CAF** & **STR** & **BUS** & **MEAN** & **PED** & **CAF** & **STR** & **BUS** & **MEAN** \\ \hline Baseline & 2.667 & 2.598 & 2.681 & 2.864 & 2.702 & 2.736 & 2.504 & 2.667 & 2.827 & 2.683 \\ LA CTC/Attention & 2.653 & 2.598 & **2.682** & 2.860 & 2.698 & 2.740 & 2.518 & 2.667 & 2.826 & 2.687 \\ SA CTC/Attention & **2.667** & **2.603** & 2.679 & **2.865** & **2.703** & **2.743** & **2.524** & **2.672** & **2.831** & **2.692** \\ \hline \hline \end{tabular}
\end{table}
Table 5: The PESQ of the SA CTC/Attention for CHIME-4 compared with the LA CTC/Attention and baseline architectures.  
As the recurrence based encoder-decoder architecture is very time-consuming during training and decoding, one future research direction is to investigate utilizing the non-recursive framework. In the meantime, incorporating the attention modeling with CTC in non-recursive framework could be explored. Another future work is to improve the performances of the baseline system.  
Methodology, validation and writing, original draft, L.W. ([PERSON]) Review, editing, and formal analysis, T.L. Review and formal analysis, L.W. ([PERSON]) Supervision and project administration, Y.Y.  
This work is partially supported by the National Natural Science Foundation of China (Nos. 11590774, 11590770), the Key Science and Technology Project of the Xinjiang Uygur Autonomous Region (No. 2016A03007-1), and the IACAS Young Elite Researcher Project (QNYC201602).  
The authors declare no conflict of interest.