# A Light-Weight Autoregressive CNN-Based Frame Level Transducer Decoder for End-to-End ASR
## 2 Models  
The end-to-end ASR of this work consists of an encoder and the proposed decoder. The encoder accepts the MFSC input of an utterance ([\(4*T_{enc},83\)]) in 10 ms steps and generates the token probabilities of the utterance ([\(T_{enc},302\)]) at each frame. Each frame proceeds in 40 ms steps. The encoder input includes 80 MFSC and 3 pitch information [19] (pitch frequency and its first and second time derivatives). The 302 tokens of the encoder output include 300 word pieces [20; 21] and two special tokens (space token, re-appearance token); the space token (\"/\") separates words and the re-appearance token (\"\(\sim\)\") represents the same token is repeated at the token level [22]. For example, \"\(/ffeee\)\(\sim\)\(\sim\)\(\sim\)\(bblelele\)/\" at the frame level is translated into \"feeble\" at the token level.: \"\(l\)\" is a word piece in this example.  
Table 2 tabulates the number of frames where the blank and the space token occur at the token probabilities of the trained CTC decoder, and this works for the Librispeech dataset, including the train-clean-100 and dev-clean. The blank token dominates the frame space by occupying around 70% of the trained CTC decoder, while the space token occupies less frame space (about 50%) in the trained model of this work.  
While the blank token supports as many consecutively same tokens as possible in the CTC decoder, the re-appearance token of this work works correctly for only up to two consecutively same tokens and misses the third consecutively same tokens. In the ground-truth token level LibriSpeech dataset used in this work, only 18 utterances have three consecutive same tokens out of the total 7,117,087 tokens (Table 3), and no utterances have four consecutively same tokens.  
\begin{table}
\begin{tabular}{c c c} \hline \hline  & **Train-Clean-100 (\%)** & **Test (\%)** \\ \hline CTC (blank) & 5,986,615 (66.7) & 1,283,175 (68.1) \\ This work (space) & 4,119,724 (45.9) & 888,293 (47.1) \\ \hline \hline \end{tabular}
\end{table}
Table 2: The number of blank tokens in CTC and space token in this work at the frame level for LibriSpeech dataset.  
The encoder shown in Figure 1, is made up of a sub-sampling CNN, an 18-layer conformer [23], and a linear layer. This encoder is used with the RNN-T decoder, the transformer decoder and the CTC decoder as well as this work, for comparison.  
The decoder shown in Figure 2 accepts the token probabilities at the frame level (encoder output) (\([T_{enc},302]\)) as input and generates the token sequence (\([U]\)) as output during the inference operation. Because the autoregressive models are more accurate than the non-autoregressive models such as CTC, the autoregressive model is adopted in this work. The autoregressive decoder of this work adds two token probabilities to generate the 302 added probabilities at each frame. One of the two token probabilities is the frame-level token probability generated from the MFSC speech input by the encoder, and the other is the token-level token probability generated from the previous token sequence. Then, the decoder selects the token with the largest added probability as the current frame at each frame and appends the current token to the output token sequence at the token transition, that is, when the current token is different from the token at the preceding frame. The decoder repeats the above operations sequentially in frame units and hence the output token sequence (\([u]\)) grows as the frame proceeds.