ProsoDeep Project Video Presentation
Channel: ProsoDeep
Date: 2019-03-19

Speech prosody is a multidimensional phenomenon comprising intonation, energy, and rhythm. It is the carrier of both linguistic information, e.g. sentence structure, focus, contrast, and lexical stress; as well as paralinguistic information, e.g. gender, age, personality, and emotions. With the recent proliferation of virtual assistants and virtual call operators, disentangling and decoding this information from speech prosody has become of great importance.

The main result of the ProsoDeep project is the creation of a state-of-the-art prosody model -- the Variational Prosody Model (VPM), which bridges the gap between traditional prosody modelling approaches, and modern black box Deep Learning based systems. The VPM can capture and reveal variation in prosodic structure within the imposed constraints of the modelling framework. Specifically, the VPM can map out a prosodic latent space representation of the prosodic variety manifold at different linguistic levels of organisation.

The VPM can be used to analyse and to synthesise prosody that corresponds to a variety of interacting communicative functions including but not limited to: attitudes, syntax, and focus. Specifically, in an automatic speech recognition (ASR) system the VPM can potentially be used to detect linguistic functions in the prosody of the input speech, while in a text-to-speech synthesis (TTS) system it can be used to  embed these communicative functions in the output. 

A secondary result of the ProsoDeep project is the recording of a prosodically rich Database, which after segmentation and annotation will be made available as open access on Zenodo. The Database has been recorded in 6 languages: English, German, Chinese, Vietnamese, Macedonian, and French, each recorded with a single native speaker. Each language comprises of 70 utterances spoken in 11 different attitudes with varied word focus, with a total of 800 utterances. The recordings were also augmented with breathing measurements and video.

The ProsoDeep project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie Actions (MSCA) grant agreement No 745802: “ProsoDeep: Deep understanding and modelling of the hierarchical structure of Prosody”.

ProsoDeep website https://gerazov.github.io/prosodeep/
ProsoDeep code https://github.com/gerazov/prosodeep