# A Light-Weight Autoregressive CNN-Based Frame Level Transducer Decoder for End-to-End ASR
## 2 Models
### Training  
The encoder and the CNN block of the proposed decoder are trained using the model shown in Figure 5, which is similar to the training model of the RNN-T decoder [18]. The loss function is minimized to match the input sequence in MFSC to the ground truth token sequence y [\(U\)]:  
\[Loss=\log\sum_{u=1}^{U}\alpha(t,u)\beta(t,u) \tag{1}\]  
Figure 4: Proposed frame-level autoregressive decoder (inference).  
Figure 3: The CNN block of the proposed decoder.  
The loss function is determined by the sum of the products of \(\alpha(t,u)\) and \(\beta(t,u)\) for all \(u\) from \(1\) to \(U\), where the product of \(\alpha(t,u)\) and \(\beta(t,u)\) represents the probability that the token at the frame \(t\) is \(y(u)\). \(y\) is the ground truth token sequence and \(U\) is the number of tokens in \(y\). \(\alpha(t,u)\) is called the forward variable and \(\beta(t,u)\) is called the backward variable [12]:  
\[\alpha(t,u)=\sum_{\pi\in B^{-1}(y),\pi_{t}=y_{u}}\prod_{t^{\prime}=1}^{t^{ \prime}=t}p_{t^{\prime},u^{\prime}}(\pi_{t^{\prime}}) \tag{2}\]  
\[\beta(t,u)=\sum_{\pi\in B^{-1}(y),\pi_{t}=y_{u}}\prod_{t^{\prime}=t+1}^{t^{ \prime}=T_{enc}}p_{t^{\prime},u^{\prime}}(\pi_{t^{\prime}}) \tag{3}\]  
\(B^{-1}(y)\) is the set of frame-level token sequences that generate the token-level ground truth \(y\) by deleting repeating tokens. \(p_{t,u}(k)\) in (2) and (3) is determined by the sum of the frame level probability \(x_{t}(k)\) from the encoder output and the token level probability \(z_{u}(k)\) from the CNN output, for a token \(k\) :  
\[p_{t,u}(k)=softmax(x_{t}(k)+y_{u}(k)) \tag{4}\]  
The forward variable \(\alpha(t,u)\) can be calculated recursively from the start \((t=1,u=1)\):  
\[\alpha(t,u)=\alpha(t-1,u-1)p_{t,u-1}(y_{u})+\alpha(t-1,u)p_{t,u}(y_{u}) \tag{5}\]  
Similarly, the backward variable \(\beta(t,u)\) can be calculated recursively from the end (\(t=T_{enc}\), \(u=U\)):  
\[\beta(t,u)=\beta(t+1,u+1)p_{t+1,u}(y_{u+1})+\beta(t+1,u)p_{t+1,u}(y_{u}) \tag{6}\]  
The recursive equations of this work (5), (6) are similar to those of CTC [12] because both decoders work at the frame level, except that the CTC decoder includes a \"Blank\" token while this work does not. The forward path computation of \(\alpha(t,u)\) was presented graphically in Figure 6 for this work, CTC and RNN-T decoders. Both forward and backward path computation equations of \(\alpha(t,u)\) and \(\beta(t,u)\) are compared in Table 4 for the preceding three decoders.  
Figure 5: The proposed frame level autoregressive decoder (training).  
The partial derivatives of Loss w.r.t. \(x_{t}\) (encoder output) and \(z_{u}\) (CNN output) are used to compute the gradients for backpropagation:  
\[\frac{\partial Loss}{\partial x_{t}}=-\frac{1}{\exp{(-Loss)}}\{\sum_{u}\frac{ \partial\alpha(t,u)\beta(t,u)}{\partial p_{t,u}(y_{u})}\frac{\partial p_{t,u}( y_{u})}{\partial x_{t}}\} \tag{7}\]  
\[\frac{\partial Loss}{\partial z_{u}}=-\frac{1}{\exp{(-Loss)}}\{\sum_{t}\frac{ \partial\alpha(t,u)\beta(t,u)}{\partial p_{t,u}(y_{u})}\frac{\partial p_{t,u} (y_{u})}{\partial z_{u}}+\sum_{t}\frac{\partial\alpha(t,u+1)\beta(t,u+1)}{ \partial p_{t,u}(y_{u+1})}\frac{\partial p_{t,u}(y_{u+1})}{\partial z_{u}}\} \tag{8}\]