## 1 Introduction  
Text-to-Speech (TTS) is a technique that takes text as an input and produces audible speech as an output. With the rapid development of machine learning over the years, the TTS technology has made remarkable progress in its goal of generating natural speech that is close to human speaking. However, there is still a huge gap in the diversity of the expressiveness of generated speech compared with natural speech. Therefore, improving the expressiveness of synthesized speech and enriching the diversity of expressiveness has become a hot topic. Speech is a hierarchical system [1] that comprises phonemes, syllables, words, phrases, and sentences. Phonemes and syllables are defined by phonetic features; words, phrases and sentences are mainly defined by semantic knowledge. Due to the hierarchical properties of speech, many studies aim to use multi-scale information to enrich the expressiveness of speech synthesis. Sentence-level style vectors are first adopted in CHiVE [2], where linguistic information of the different scales is introduced to both encoding and decoding networks of a variational autoencoder (VAE). However, single scale style representation is not expressive enough. Researchers focus on using multi-scale style vectors later. Google [3] introduced phoneme and word-level latent variables to Tacotron2 to enhance generation diversity by sampling. HiMuV-TTS [4] investigated phoneme and sentence-level variables on FastSpeech. Moreover, the extracted style vectors are semantically related to the text. Randomly sampling from latent variables leads to unnatural prosody. Therefore, MsEmoTTS [5] was proposed to predict syllable and sentence-level style vectors from text for inference. [PERSON] et al. [6] subsequently used hierarchical textual information to predict multi-scale prosody information. However, there still remain some problems in the existing research on multi-scale style modeling. First, the style vectors of different scales are usually extracted in parallel. The dependence of different scale style vectors is not considered. Second, the style extraction and style prediction modules are not jointly optimized, which leads to a mismatch between training and inference. Finally, previous work on style has modeling mainly focused on acoustic models, resulting in poor sound quality. The frame, as the basic unit of speech acoustic features, is also an important scale level in the speech generation process because it connects phonetic information to the speech waveform signal generation process. New models, VITS and NaturalSpeech [7,8], utilize frame-level variables to implement end-to-end training of acoustic models and vocoder, which significantly improves synthetic sound quality  
We believe that, for a specific speech segment, style vectors of different scales all describe the same style pattern, and the main difference is that coarser-grained vectors depict changes over a longer period of time, and finer-grained vectors enrich more detail. Therefore, in the process of style extraction [1], fine-grained style features should be extracted first, and then coarse-grained information is abstracted on top of the fine-grained information. The process of predicting is just the opposite [9]. First comes generating coarse-grained information to set the overall tone. Then, more fine-grained style information is predicted based on the coarse-grained style. In this way, styles of different scales can be coordinated and organized. We can view linguistic, phonological, and acoustic features as the representations of speech at different scales. Therefore, we propose an expressive end-to-end text-to-waveform generation model called HierTTS. It considers the latent variables of frames, phonemes, syllables, words, and sentences and achieves highly expressive speech generation by mining multi-scale information from both modalities, text and speech. In addition, we employ a Gaussian distribution to model the latent variables at each scale. Gaussian sampling helps to enrich generation diversity.