# A Light-Weight Autoregressive CNN-Based Frame Level Transducer Decoder for End-to-End ASR
## 3 Experiments and Results  
The CNN block of the proposed decoder was trained along with the encoder by using the 100 h Librispeech [24] clean dataset (train-clean-100). To avoid overfitting, data augmentation techniques such as speed perturbation [26] and SpecAugment [27] were used for the training. For the speed perturbation, training was performed three times for each dataset, one at 0.9\(\times\), another at 1.0\(\times\) and the other at 1.1\(\times\) speed. For the SpecAugment, two frequency masks and two time masks were applied to every input utterance along with the time warping. Each frequency mask ranges a consecutive range of up to 30 frequency bins. Each time mask ranges from consecutive time frames up to 40 frames. The time warping shifts the time frame up to five frames toward left and right. The 300 word pieces generated from SentencePiece [20] and two special tokens were used in this work to represent the token sequence. The special tokens include the space token and the re-appearance token. The space tokens are used for no-token frames or to separate words in this work, while the blank tokens of the CTC decoder are used to separate tokens or for no-token periods. The re-appearance token is used for the frames where the same token is repeated in the token sequence. The Adam optimizer [23] was used for the training \begin{table}
\begin{tabular}{l l l l} \hline \hline
**Decoder Type** & **Forward Variable \(\boldsymbol{a(t,u)}\)** & **Backward Variable \(\boldsymbol{a(t,u)}\)** \\ \hline \multirow{3}{*}{CTC} & wordpiece (even \(u\)) & \((\alpha(t-1,u-2)+\alpha(t-1,u-1)\) & \((\beta(t-1,u-2)+\beta(t-1,u-1)\) \\  & \(+\alpha(t-1,u))*p_{t}(y_{u})\) & \(+\beta(t-1,u))*p_{t}(y_{u})\) \\ \cline{2-4}  & blank (odd \(u\)) & \((\alpha(t-1,u-1)+\alpha(t-1,u))*p_{t}(y_{u})\) & \((\beta(t-1,u+1)+\beta(t-1,u))*p_{t}(y_{u})\) \\ \hline \multirow{2}{*}{RNN-T} & \(\begin{array}{l}\alpha(t-1,u)*p_{t-1,u}(blank)\\ +\alpha(t,u-1)*p_{t-1,u}(y_{u})\end{array}\) & \(\begin{array}{l}\beta(t+1,u)*p_{t,u}(blank)\\ +\beta(t,u+1)*p_{t,u}(y_{u})\end{array}\) \\ \hline \multirow{2}{*}{This work} & \(\begin{array}{l}\alpha(t-1,u-1)*p_{t+1,u-1}(y_{u})\\ +\alpha(t-1,u)*p_{t,u}(y_{u})\end{array}\) & \(\begin{array}{l}\beta(t+1,u+1)*p_{t+1,u}(y_{u+1})\\ +\beta(t+1,u)*p_{t,u}(y_{u})\end{array}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Forward/Backward algorithm of decoders for end-to-end ASRs. Figure 6: Forward path computation of \(\alpha(t,u)\) of decoders for end-to-end ASR. with the hyperparameters of \(\beta_{1}=0.9,\beta_{2}=0.98,\epsilon=10^{-9}\). Also, the Noam learning rate scheduling [9] was used with the 25k warmup steps and the learning rate factor of 5.0. The model was trained up to 120 epochs, and the trainable parameters of the last 10 epochs were averaged to obtain the inference model. The kernel size is 15 for the CNN blocks of the 18-layer conformer comprising the encoder block shown in Figure 1.  
To assess the performance of this work, the WERs and the numbers of trainable parameters of this work were compared in Table 5 with the non-autoregressive decoders (CTC [12], self-conditioned CTC [17], intermediate CTC [16]) and the autoregressive decoders (transformer [9], RNN-T [18]); the same encoder with 30M trainable parameters was used in this comparison. This work gives better WERs compared to the non-autoregressive decoders and comparable WERs to the autoregressive decoders while the number of trainable parameters is much smaller than those of the autoregressive decoders (177 k in this work versus 10 M in the transformer and 1 M in RNN-T). To train the RNN-T decoder, the dropout rate was 0.3 for the encoder and 0.2 for the decoder. To train the 6-layer transformer decoder, the dropout rate was reduced to 0.1 to alleviate the influence of the dropout on the attention operation, and the CTC was included in the loss function to enhance accuracy [28; 29]. In this work, the dropout rate was 0.1 for the encoder and 0.3 for the decoder. The training codes were based on the ESPNet [30].