State-of-the-Art Speech Recognition Using Multi-Stream Self-Attention\n With Dilated 1D Convolutions
Journal: arXiv (Cornell University)
Year: 2019

Self-attention has been a huge success for many downstream tasks in NLP,\nwhich led to exploration of applying self-attention to speech problems as well.\nThe efficacy of self-attention in speech applications, however, seems not fully\nblown yet since it is challenging to handle highly correlated speech frames in\nthe context of self-attention. In this paper we propose a new neural network\nmodel architecture, namely multi-stream self-attention, to address the issue\nthus make the self-attention