# A Review of Deep Learning Based Speech Synthesis
## 4 Deep Learning Based Speech Synthesis
### Multi-Distribution Deep Belief Networks for Speech Synthesis  
The multi-distribution deep belief network (DBN) [43] is a method of modeling the joint distribution of context information and acoustic features. It models the coutinuous spectral, discrete voiced/unvoiced (V/UV) parameters and the multi-space F0 simultaneously with three types of RBMs. Due to the different data types of the 1-out-of-\(K\) code, the F0, the spectral and the V/UV parameters, the method uses the 1-out-of-\(K\) code of the syllable and its corresponding acoustic parameters as the visible-layer data of the RBM to train the RBMs. In DBNs, the visible unit can obey different probability distributions; therefore, it is possible to characterize the supervectors that are composed of these features. In the training phase, given the 1-out-of-\(K\) code of the syllable, the network fixes the visible-layer units to calculate the hidden-layer parameters firstly, and then uses the parameters of the hidden layers to calculate the visible-layer parameters until convergence. Finally, the predicted acoustic features are interpolated based on the length of the syllable.  
\begin{table}
\begin{tabular}{l l l} \hline \hline
**Methods** & **Advantages** & **Disadvantages** \\ \hline HMM & Flexible with changing voice characteristics and the system is robust & The acoustic features are oversmoothed, making the generated speech sounds muffled \\ RBM & Can better describe the distribution of high-dimensional spectral envelopes to alleviate the over-smooth problem & Suffer from the fragementation problem of training data \\ DBN & Cannot suffer from the training data fragementation problem and reduce the over-smoothing problem & The quality of generated speech \\ DMDN & Can solve the single modality problem & will be degraded \\ DBLSTM & Can fully leverage contextual information & Can only leverage limited contexts and each \\ WaveNet & Can produce high-quality speech waveforms & frame is mapped independently \\ Tacotron & Fully end-to-end speech synthesis model and can produce high-quality speech waveforms & Still needs a vocoder to synthesize waveform \\ CNN & Fast to train the model & The speech quality might be degraded \\ \hline \hline \end{tabular}
\end{table}
Table 1: The advantages and disadvantages of different speech synthesis methods, including hidden Markov model (HMM), restrictive Boltzmann machine (RBM), deep belief network (DBN), deep mixture density network (DMDN), deep bidirectional long short-term memory (DBLSTM), WaveNet, Tacotron and convolutional neural network (CNN).  
The advantage of this method is that all the syllables are trained in the same network, and all the data are used to train the same RBM or DBN. Therefore, it cannot suffer from the training data fragementation problem. In addition, modeling the acoustic feature parameters of a syllable directly can describe the correlation of each frame of the syllable and the correlation of different dimensions of the same frame. The method avoids averaging the frames corresponding to the same syllable, thus reducing the over-smooth phenomenon. However, since this method does not distinguish syllables in different contexts, it still averages the acoustic parameters corresponding to the same syllable. In addition, compared to the high-dimensional spectral parameters, the one-dimensional F0s don't contribute much to the model, thus making the predicted F0s contain a lot of noise that reduces the quality of the synthesized speech.