# Gated Recurrent Attention for Multi-Style Speech Synthesis
## 3 Gated Recurrent Attention  
The LSA aligns a sequence of the decoder states with the corresponding sequence of the encoder outputs, conditioned on the previous recurrent state at each decoder time-step. The mechanism of updating the recurrent state in the LSA is similar to that of updating the hidden state in the vanilla RNN as shown in Figure 2a,c. Similar to Equation (5), where the hidden state is updated through a function of the content \(x_{n}\) and the previous hidden state \(h_{t-1}\), the recurrent state \(f_{n,t}\) in LSA is updated by filtering the normalized score of content \(x_{n}\) and the previous recurrent \(f_{n,t-1}\) in Equations (7)-(9). As described in [34], although the vanilla RNN has a powerful ability to represent context, due to the problem of vanishing gradient, it is inadequate for the tasks involving long-term dependencies. Likewise, we can assume that the LSA is not suitable for reflecting long-term contexts. For e2e speech synthesis, it is important to deal with the **long-term contexts** in an attention model. Other tasks using attention-based seq2 seq models such as machine translation [25] and speech recognition [21] generate a sequence of characters or words. However, for speech synthesis, the attention-based decoder generates a speech sequence which is usually much longer than the input sequence of characters or words. Analogous to the vanishing gradient problem of the vanilla RNN, as the length of the decoder state sequence increases, the simple recursive structure in LSA may deteriorate the performance of the attention. Moreover, it is necessary for an attention model to be able to control the degree of **updating the recurrent state**\(f_{n,t}\). The duration and acoustic characteristics of each character in spoken utterances differ by the character, utterance, and context. A vowel is a short or long monophthong or diphthong according to the context, e.g., a vowel \(i\) sounds [i] in _bit_, [i] in _police_, or [a] in _bite_. To model the intra-character and inter-character phoneme transitions in various prosody, the recurrent state which contains contextual information should be updated faster near the transition-boundary than far from the transition-boundary. Furthermore, the **location-sensitivity** is important for speech synthesis. The score's sensitivity to the recurrent state is crucial for generating not only the alignment but also the spectrogram. The decoder state at the \(t\)-th step \(y_{t}\) is generated by attending encoded inputs \(x\) corresponding to the score \(e_{n,t}\) as in Equations (8)-(11). The decoder state sequence is used to synthesize the output mel-spectrogram in Tacotron. From a single character, several frames of spectrograms which have different spectral characteristics should be generated in speech synthesis. In speech recognition, the encoder encodes the spectral characteristics which have positional dependencies. However, in speech synthesis, the encoder output merely specifies the character to be synthesized. In other words, to generate the decoder state \(y_{t}\), it is necessary to know where the frame is located by conditioning the previous recurrent state \(f_{n,t-1}\). Thus, for e2e speech synthesis, an attention mechanism that can control the **location-sensitivity** and **updating the recurrent state** while considering a **long-term context** is required. Figure 2: Illustration of recurrences in (**a**) Vanilla recurrent neural network (RNN), (**b**) Gated recurrent unit (GRU), (**c**) Location-sensitive attention (LSA), and (**d**) Gated recurrent attention (GRA). We propose an attention mechanism which can model and generate alignment between two sequences with style-variability and long-term dependency. It includes an update gate \(z_{t}\) and a scoring gate \(r_{t}\), which are presented respectively by  
\[z_{t}=\sigma\left(W_{z}s_{t-1}+V_{z}x_{n}+U_{z}f_{n,t-1}+b_{z}\right), \tag{13}\]  
\[r_{t}=\sigma\left(W_{r}s_{t-1}+V_{r}x_{n}+U_{r}f_{n,t-1}+b_{r}\right) \tag{14}\]  
where \(\sigma\) is a logistic sigmoid function.