# A Light-Weight Autoregressive CNN-Based Frame Level Transducer Decoder for End-to-End ASR
## 1 Introduction
(Table 1) compares the characteristics of the above-mentioned decoders used for end-to-end ASR. The transformer decoder generates the token probabilities at the token level from the encoder output and the previous token sequence by using the transformer blocks. Since the number (\([U]\)) of these token probabilities is the same as the number (\([U]\)) of the ground truth tokens, the transformer decoder can calculate the loss by using the cross entropy without any special tokens such as blank tokens. Only two special tokens(\(<\)sos\(>\), \(<\)eos\(>\)) are employed in the transformer decoder to align the attention blocks at the start and end points of an input utterance. The CTC decoder generates the token probabilities at the frame level which are the encoder output. Since the number (\([T_{enc}]\)) of these token probabilities is not the same as the number (\([U]\)) of the ground truth token probabilities, the CTC decoder cannot use the cross entropy but uses the forward/backward algorithm for the loss computation. The forward/backward algorithm adds \"blank\" tokens as token separators and at non-token frames for training with the mismatch between \(T_{enc}\) and \(U\). The RNN-T decoder works similarly to the CTC-decoder except that it generates the token probabilities at the frame + token level from the encoder output and the previous token sequence. Thus, the number of these token probabilities is \(T_{enc}+U\). This work generates the token probabilities at the frame level to reduce the decoding time. Since the number (\([T_{enc}]\)) of these token probabilities is not the same as the number \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Autoregressive** & **Decoding Level [Length]** & **Special Tokens** & **Training Algorithm** \\ \hline Transformer [9] & O & token level \([U]\) & \(<\)sos\(>\), \(<\)eos\(>\) & Cross-Entropy \\ CTC [12] & X & frame level \([T_{enc}]\) & blank & Forward/backward \\ RNN-T [18] & O & Token + Frame \([U+Tenc]\) & blank & Forward/backward \\
**This work** & **O** & **frame level \([T_{enc}]\)** & **space, re-appearance** & **Forward/backward** \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Comparison of decoders for end-to-end ASR. ([_UJ_]) of the ground truth token probabilities, the forward/backward algorithm is used in this work, as in the CTC and RNN-T decoders. The word separator token (\"space token\") is used in this work instead of the blank token to avoid the deletion of short-duration tokens in the CTC decoders. Since the blank token is inserted at every token boundary and occupies a frame time in the CTC decoder, a short-duration token that lasts only one or two frames is deleted at the CTC decoder output. Because the space token of this work occurs less frequently than the blank token, the short-duration token is less probable to be deleted in this work. The blank token occupies around 70 percent of frames in the trained frame level token sequence of the CTC decoder while the space token occupies around 50 percent of frames in the trained frame level token sequence of this work. The re-appearance token is introduced in this work to replace the role of the blank token which keeps the consecutive-same token sequence. In this work, an autoregressive decoder without using the beam search is proposed for end-to-end ASR; the autoregressive architecture enhances accuracy and the non-use of the beam search reduces the computation time. To further reduce the computation time, two schemes are proposed in this work. One is to add the token probabilities at the frame level ([_TencL_]) as mentioned before. The other is the use of a small-size CNN in the decoder feedback loop. Compared to the RNN-T and the transformer decoders, this work achieved comparable accuracy with half and \(1/30\) decoding times, respectively. Section 2 explains the proposed CNN-based frame level autoregressive decoder for end-to-end ASR. Section 3 presents the experimental results. Section 4 discusses this work.