# Gated Recurrent Attention for Multi-Style Speech Synthesis
## 1 Introduction
Several embedding methods have been proposed to represent styles for e2e speech synthesis: an embedding lookup to give speaker identity as a condition [18], an embedding vector from a reference encoder for modeling the style of an utterance [2; 4], and a speaker embedding trained for speaker verification [19]. The style embedding vector is concatenated with the output of the text encoder in the e2e speech synthesis systems. The reference-encoder-based techniques [2; 4; 20] have shown that a speech with similar prosodic characteristics can be generated by conditioning the e2e speech synthesis model with an unsupervised style representation derived from a reference utterance. However, when synthesizing speech with styles rarely found in the training data, the prosodic similarity between the reference speech and the synthesized speech tends to be low. Among various factors of the prosody, it is more difficult to model the local style such as rhythm or duration, rather than the global style which is usually determined by tone or speaker identity. It has been reported that the speaker similarity observed from the speech synthesized with unseen speaker-style is low although the naturalness is good [19]. One of the reasons for these problems is that the attention-update at each decoder step is too simple to cover various styles in a multi-speaker database. Such simplicity causes the attention-based decoder to be overfitted to the styles that have been seen during the training session. For these reasons, in order to learn and synthesize voices of various styles, an attention mechanism that can preserve longer-term context and adjust the duration depending on the context is highly required. In this paper, we propose a novel attention model based on gated recurrence, which we call gated recurrent attention (GRA). GRA controls the contextual information by employing two gates: the update gate and the scoring gate. The update gate determines how much the attention forgets the previous recurrent state which retains the attention alignments from the previous steps. The scoring gate determines how much the previous recurrent state is reflected to derive the attention alignment and decoder output. The two gates are computed according to the attention key, query, and the previous recurrent state. The GRA's recurrent states are the convolved attention weights, which are also known as the location [21]. The recurrent attention (RecAtt) and recurrent neural network (RNN) attention (RNNAtt) [22] apply a vanilla RNN to weighted input embeddings, which are also known as context [22] or glimpse [23]. Gated recurrent unit (GRU)-gated attention (GAtt) [24] uses a GRU to generate a recurrent representation of the input embeddings depending on the previous decoder state. GRA differs from the three other attention methods in that the GRA seeks attention alignments according to where it was, not to what was, focused in the previous time-step. For this reason, GRA is useful for tasks that generate different outputs depending on the location even if the inputs are the same, such as speech synthesis. GRA can be regarded as a gated recurrent version of the location-sensitive attention (LSA) [21] as the gates in GRA operate similarly to the gates in the GRU [25]. GRA is expected to be advantageous for retaining long short-term information and creating shortcuts between multiple temporal steps in a similar fashion to the GRU [26]. GRA has been found useful for seq2 seq tasks to deal with long sequences where there exists huge variability in the alignments between the input and output sequences.