# An Improved Chinese Pause Fillers Prediction Module Based on RoBERTa
## 2 Related Works
### Tts  
TTS is a technology that converts text into speech. It takes input text and transforms it into audible speech output using speech synthesis techniques [10; 11; 12]. TTS systems find wide applications in speech synthesis, accessibility technologies, education, and various other fields. A typical TTS system consists of two main components: the front-end and the back-end. The front-end is responsible for analyzing the input text and extracting linguistic information and speech features, and then generating an intermediate representation in the form of a phoneme sequence. The back-end utilizes these phoneme sequences and acoustic models to generate the final speech output using speech synthesis algorithms.  
In a TTS system, the pause fillers prediction module plays a crucial role [13]. It automatically determines and inserts appropriate pause fillers based on the input text and contextual information, aiming to enhance the naturalness and fluency of the synthesized speech. By incorporating the pause fillers prediction module, the TTS system can better simulate the pauses and thinking processes in human speech expression, making the synthesized speech more realistic and easier to comprehend.  
In recent years, more and more TTS systems have been used in Chinese. For example, Tacotron2 system, parallel WaveNet system, Fastspeech system, and so on. [PERSON] et al. proposed Myanmar TTS synthesis [14] using an end-to-end model. [PERSON] et al. used text context embedding to directly predict the prosodic features of reference audio, and text context embedding is calculated by BERT (a pre-trained language representation model). [PERSON]'s improved system could generate richer prosodic speech during the inference stage with limited training data [15]. [PERSON] and [PERSON] proposed a speech synthesis model based on fine-grained style representation, called word-level style variation (WSV) [16]. In order to improve the accuracy of WSV prediction and the naturalness of synthesized speech, [PERSON] and [PERSON] used a pretrained BERT model and speech information to derive semantic descriptions. [PERSON] et al. proposed a speech synthesis method [17] based on LPCNet. [PERSON] et al. proposed an end-to-end speech synthesis method based on WaveNet [18]. [PERSON] and [PERSON] designed two context encoders [19], the sentence window context encoder and the paragraph-level context encoder. The context representation is extracted by BERT from multiple sentences through an additional attention module. They also proposed a deep learning method using BERT to provide a wide range of contextual representations for statistical parametric speech synthesis (SPSS) [20; 21; 22]. In order to explore the problem of zero-shot TTS, [PERSON] et al. proposed a speaker-conditional architecture [23] that includes a flow-based decoder. The speaker-conditional architecture achieved state-of-the-art results for similarity with new speakers using video of only eleven speakers. [PERSON] et al. developed Ada-Speech 4 [24], a zero-shot adaptive TTS system for high-quality speech synthesis. It could achieve better voice quality and similarity than baselines in multiple datasets without any fine-tuning. [PERSON] et al. presented a novel zero-shot multi-speaker speech synthesis approach (ZSM-SS) [25]. Compared to the normalization architecture, ZSM-SS added non-autoregressive multi-head attention between the encoder-decoder architecture [26; 27; 28].