# Gated Recurrent Attention for Multi-Style Speech Synthesis
## 1 Introduction  
It is crucial to produce natural prosody which includes tone, intonation, stress, and rhythm to generate a natural speech utterance. In the context of acoustics or speech signal processing, the prosody is interpreted as the composition of the duration of each phoneme in the utterance, the pitch contour over the whole sentence, and the spectral characteristics including amplitude. Each of these elements can be analyzed and synthesized individually, or jointly, depending on the modeling methods [1, 2]. The prosody can be affected by various factors including identity of the speaker, emotion of the speaker, purpose of the speech, form of the speech, how lexical items relate to each other semantically or syntactically, and where the speaker places emphasis [3]. Styles are common prosodic characteristics in an utterance or several utterances, affected by shared factors in the speech. For example, we designate a style of a particular speaker identity with some common prosodic attributes observed from the utterances of the speaker. In the field of speech synthesis, techniques for modeling or generating styles have been studied [1, 2, 4]. In the statistical parametric speech synthesis (SPSS) studies [5, 6, 7], the speech is synthesized by first estimating the duration of each phoneme or phoneme state according to the context, and then regressing acoustic features such as the pitch and spectral information for the estimated duration. Although explicit statistical modeling of duration and acoustic features followed by speech parameter generation improves robustness against data sparseness, it degrades the quality and naturalness of the synthesized speech mostly due to the oversmoothed parameter trajectories [6]. In other words, the prosodical information may be discarded in the statistical modeling stage and cannotbe recovered in the synthesis stage by generating speech parameters merely from the means and variances of acoustic features. In order to compensate for the loss during statistical modeling, SPSS studies have used adaptation techniques, such as maximum likelihood linear regression (MLLR) [1; 8], factored MLLR [9], learning hidden unit contribution (LHUC) [10; 11], or retraining with control vector [12], to adapt to the style of various speakers and emotions. Though successful for some specific application [1; 11], these techniques still suffer from oversmoothing, poor adaptation performance with less data, and difficulty in reflecting the style of an individual sentence. Furthermore, the duration and acoustic features are usually adapted separately, which can be considered undesirable for natural style expression. Another shortcoming of these approach is that they require labeled data for supervised training of the adaptation model. Most of the aforementioned shortcomings in style modeling for SPSS can be somewhat mitigated in end-to-end (e2e) speech synthesis models. The e2e speech synthesis systems [13; 14; 15] based on sequence-to-sequence (seq2 seq) models with attention [16; 17] aim to synthesize the speech directly from a sequence of characters. The seq2 seq model for speech synthesis consists of an encoder, an attention-based decoder, and a postprocessing network. The encoder embeds a sequence of characters into a sequence of embedding vectors. The attention-based decoder generates a stream of intermediate spectral features such as mel-spectrogram by attending on the character embedding vectors extracted from the input texts. The postprocessing network converts the intermediate spectral features into spectrograms with larger dimensionality. Unlike SPSS, which explicitly estimates the duration of each phoneme state and generates the acoustic parameters according to that duration, the e2e speech synthesis framework implicitly determines the duration by updating the decoder hidden state through an attention mechanism. This implicitly determined duration enables to apply a style that affects both the duration and spectral features with a single vector of style representation.