# A Light-Weight Autoregressive CNN-Based Frame Level Transducer Decoder for End-to-End ASR
## 1 Introduction  
Currently, end-to-end automatic speech recognition(ASR) methods are widely used to extract text from the time-domain speech input [1; 2; 3], because the entire end-to-end ASR block can be trained by using a deep learning network. The end-to-end ASR block consists of an encoder and a decoder. For example, the encoder converts the mel-frequency spectrum coefficient (MFSC) speech input ([4 \(*T_{enc}\), 83]) to an intermediate output ([\(T_{enc}\), 302]); \(4*T_{enc}\) is the number of frames of the input speech sentence. Each frame has an overlapped window of 25 ms long and proceeds in 10 ms steps. The encoder output ([\(T_{enc}\), 302]) represents the probabilities of 302 tokens in 40 ms steps; the 302 token includes 300 word pieces and two special tokens. The decoder converts the encoder output to a text ([\(U\)]) at the token level; \(U\) is the number of word pieces corresponding to the encoder input. The encoder block of the end-to-end ASR can be implemented with various deep learning networks, such as convolutional neural network (CNN) [4; 5; 6], long short term memory (LSTM) [7; 8] or transformer [9; 10; 11]. The decoder block can be implemented either in autoregressive or non-autoregressive methods. The autoregressive decoder accepts the previous tokens along with the encoder output as the decoder input. The non-autoregressive decoder accepts only the encoder output as the decoder input. The connectionist temporal classification(CTC) [12] is one of the representative non-autoregressive decoders; it generates a loss function to train the end-to-end ASR which generates a text at the token level from the speech input in the frame level. The frame level refers to the time domain, that is, variables in the frame level can change values in unit time intervals, in this work, in 40 ms intervals. The token level refers to the text domain, that is, variables in the token level can change values at the token transition. The CTCdecoder does not include any trainable neural network blocks. The CTC performs two functions. One is the conversion of the encoder output (\([T_{enc},301]\)) to the text (\([U]\)). The CTC selects a max-valued index out of 301 tokens per frame to generate an intermediate token sequence (\([T_{enc}]\)) and eliminates the blank tokens and the consecutively duplicate tokens from the intermediate token sequence to generate the \([U]\) text. The blank token represents a 'non-token' frame or a 'token transition' frame. The other is to generate a loss function to train the encoder block by accumulating all the matching probabilities between the intermediate token sequence (\([T_{enc}]\)) and the ground truth text (\([U]\)) along both the forward(previous) and the backward(future) directions. Compared to the autoregressive decoders, the CTC-based decoders [13; 14; 15; 16; 17] have merits of much less computation time with less accuracy, because there are no trainable parameters and the parallel computation is enabled by no feedback operation. The representative autoregressive decoders for end-to-end ASR are the transformer decoder [9] and the RNN transducer(RNN-T) decoder [18]. The RNN-T decoder is similar to the CTC decoder except that it adds the processed previous token sequence (\([U,300]\)) and the encoder output (\([T_{enc},300]\)) to generate the decoder input (\([U+T_{enc},300]\)). An LSTM block is included in the RNN-T decoder to generate the processed token sequence from the previous token sequence (\([U]\)). The transformer decoder includes a transformer block to convert the encoder output (\([T_{enc},256]\)) to the probabilities of the output token sequence (\([U,302]\)); this enables the use of a simpler loss function of cross entropy. However, the transformer decoder suffers from long computation time because it cannot employ parallel computation due to the feedback operation and the relatively large number of trainable parameters. Besides, the accuracy is degraded in the transformer decoder without using the additional beam search.