Design and Evaluation of Statistical Parametric Techniques in Expressive Text-To-Speech: Emotion and Speaking Styles Transplantation

Together with the rise in the implementation of technology in daily life the awareness of the usefulness of speech technology as a nexus between humans and machines has risen too. Concretely in the field of text to speech conversion there have been huge advancements in terms of naturalness and quality spurred by big investments carried out by some of the most successful multinational technological companies. Most of the advances have come thanks to the progress made in techniques based on deep neural networks, capable of generating voices of very high quality in traditional applications. Nonetheless, designing expressive text to speech synthesis systems with high naturalness is still a problem mainly because of data scarcity. Generally obtaining good synthesizers require vast amounts of information, something that is reasonably easy to obtain for neutral, read speech, but significantly harder to obtain when thinking about particular expressiveness such as emotions or speaking styles. Traditionally, the way to solve this limitation has been the application of adaptation-based techniques that convert robust models trained with diverse and abundant data into models for the desired speaker and expressiveness, but those approaches also have their limits. The present document aims to solve some of the existing problems in the traditional expressive speech synthesis techniques by focusing in one particular aspect: scalability. On one hand, gathering data for every expressiveness and speaker is a task that cannot realistically be tackled. On the other hand, traditional solutions typically induce limitations in naturalness or expressive intensity. This thesis presents first of all a system capable of controlling the paralinguistic properties of speech (the dynamic changes in the way of speaking that convey expressive information), allowing us to control the expressive strength produced by the synthesis. Thus, it becomes possible to generate different levels of expressive intensity even if the source data does not contemplate it. Secondly, it presents a system for transplanting the paralinguistic information into a different target speaker which allows us to imbue expressive information into a target speaker for whom no expressive information is available. Finally it introduces a complete and automatic expressive text to speech system, capable of detecting the most adequate expressiveness for conveying the text to be synthesized and then generate the corresponding audio file without any further interaction from the user. All the proposed systems and their variants have been perceptually evaluated thanks to the collaboration of numerous volunteers, which has allowed us to certify the benefits that our systems provide to the field of expressive speech synthesis. The evaluations were designed to minimize evaluation bias and were analyzed with strict statistical significance criteria, fixing a 95% confidence band. The results of the evaluations proved that the paralinguistic features control system is capable of controlling in a practically linear fashion the produced emotional strength (correlation coefficient of 0.97 when compared with the ideal linear system) while providing a 70.6% of emotion identification rates. These results are significantly better to those achievable by traditional techniques. In terms of the transplantation system, we see how it is capable of increasing by 1.2 points in the MOS scale the produced expressive strength of the target speaker when compared to the non transplanted version at a cost of just 0.4 points in speech quality. This translates into an average preference rate for the transplanted system of 87% when compared with a neutral system. The results are consistent when transplanting speaking styles such as live sports broadcasts or political speech. The system was also modified to add cross-lingual transplantation capabilities, which worked reasonably well when transplanting Spanish paralinguistic information into an English speaker. Finally, for the evaluation and design of the complete expressive text to speech system we began by recording a single speaker Spanish speaking styles database, SSS. This database is the first of is kind to be designed explicitly with speech synthesis in mind, a task for which using data for a single speaker significantly helps increase quality. With these data we evaluated three approaches to the speech synthesis system: a discrete one in which the most adequate speaking style is predicted from the genre of the text to be synthesized, a semi-continuous approach that improves on the discrete system by adding expressive intensity information and finally a continuous system that does not rely on gender prediction techniques and directly makes use of lexical information to generate the speech. The results show how all three systems are capable of significantly improving traditional speech synthesis systems in terms of adequacy (average improvements between 1.0 and 1.5 depending on the system) and in terms of produced expressive strength (improvements between 1.1 and 2.0 in this case). In terms of speech quality both the discrete and the semi-continuous system are capable of maintaining neutral speech synthesis rates (averages of 4 points in the MOS scale). The continuous system, mainly due to the semi-continuous nature of the training data, introduces artifacts in the synthetic speech that harm the produced speech quality.