## 3 Methods
### Loss Function  
#### 3.2.1 Joint CTC-Attention Loss  
(1) \(\mathcal{L}_{\text{CTC}}\)  
CTC is well suited for sequence modeling, and its training samples do not need to be aligned. By introducing a blank placeholder \(\varepsilon\), the problem of the alignment of input and output labels in speech recognition is solved. The core idea of CTC is to allow repeated characters and blank characters. Given an input speech sequence \(x=(x_{1},\,\dots,\,x_{T})\) and the corresponding tag sequence \(y=(y_{1},\,\dots,\,y_{U})\), the tag sequence becomes \(y^{\prime}=(y_{1},\,\dots,\,y_{U})\cup[\varepsilon]\). For example, if \(y=(d,\,o,g)\), then \(y^{\prime}=(\varepsilon,\,d,\,\varepsilon,\,o,\,\varepsilon,\,g,\,\varepsilon)\). A CTC training model to maximize \(P(y\,|\,x)\), namely in all possible tags sequence \(\Phi(y^{\prime})\) on the probability distribution of:  
\[P(y|x)=\sum_{\pi\in\Phi(y^{\prime})}P(\pi|x) \tag{1}\]  
CTC is usually applied on top of the Recurrent Neural Network (RNN). Each RNN output cell is interpreted as the probability that the corresponding label will be observed at a particular time. Tags sequence \(P(\pi\,|\,x)\) probability output is independent of the network and is modeled as a condition of the product of the:  
\[P(\pi|x)\approx\prod_{t=1}^{T}P(\pi_{t}|x)\,=\prod_{t=1}^{T}q_{t}(\pi_{t}) \tag{2}\]  
where \(q_{t}(\pi_{t})\) represents the softmax of \(\pi_{t}\) label in the output layer \(q\) of the RNN at time \(t\). The loss of CTC is minimized by the negative log-likelihood value of the true character sequence \(y^{\prime}\):  
\[\mathcal{L}_{\text{CTC}}\triangleq-\ln P(\mathbf{y}^{*}\mid\mathbf{x}) \tag{3}\]  
The probability distribution \(P(y\,|\,x)\) could be calculated before and after using the algorithm effectively:  
\[P(y|x)=\sum_{u=1}^{|y^{\prime}|}\frac{\alpha_{t}(u)\beta_{t}(u)}{q_{t}(y^{ \prime}_{u})} \tag{4}\]  
where \(\alpha_{t}(u)\) is a forward variable, representing the total probability of all possible prefixes (\(y^{\prime}_{1:u}\)) ending with the \(u\)-th label, and \(\beta_{t}(u)\) is a backward variable of all possible suffixes  
Figure 2: JSUM. (y\({}^{\prime}\)\({}_{i:U}\)) beginning with the \(u\)-th label. The network is then trained with standard backpropagation by taking the derivative of the loss function with respect to \(q_{i}(k)\) for any \(k\) labels, including whitespace characters. (2) \(\mathcal{L}_{Att}\)  
Unlike the CTC method, the attention model predicts each of the targets directly, without the need for intermediate representation or any hypothesis, and based on the probability chain rule according to the following recursion equation, directly estimates the a posteriori probability \(P(y\,|\,x)\):  
\[P(y|x)=\prod_{u}P(y_{u}|x,y_{1:u-1}) \tag{5}\]  
\[h=Encoder(x) \tag{6}\]  
\[y_{u}\,\sim\,\text{AttentionDecoder}(h,y_{1:u-1}) \tag{7}\]  
The framework of the attention mechanism consists of two RNN layers: an encoder and an attention decoder, so it can learn sequences of two different lengths based on cross-entropy criteria. The encoder converts \(x\) to a high-level representation \(h=(h_{1},\dots,h_{L})\) in Equation (6). Then, the AttentionDecoder generates a probability distribution over the character \(y_{u}\) on \(h\) and all the characters seen previously, \(y_{1:u-1}\) in Equation (7). L (L < T) is the number of frames output by the encoder. Here, a special start of the sentence (sos)/end of the sentence (eos) flag is added to the target set so that when (eos) is issued, the decoder completes the generation of the hypothesis. The loss function of the attention mechanism is computed from Equation (5) as follows:  
\[\mathcal{L}_{\text{Att}}\triangleq-\ln P(y^{*}\mid\mathbf{x})=-\sum_{u}\ln P \big{(}y_{u}^{*}\mid\mathbf{x},y_{1:u-1}^{*}\big{)} \tag{8}\]  
where \(y^{*}_{1:u-1}\) is the correct value of the previous character.