Patent Description:
Text-to-speech (TTS) synthesis methods and systems are used in many applications, for example in devices for navigation and personal digital assistants. TTS synthesis methods and systems can also be used to provide speech segments that can be used in games, movies or other media comprising speech.

There is a continuing need to improve TTS synthesis systems. In particular, there is a need to improve the quality of speech generated by TTS systems such that the speech generated retains vocal expressiveness. Expressive speech may convey emotional information and sounds natural, realistic and human-like. TTS systems often comprise algorithms that need to be trained using training samples and there is a continuing need to improve the method by which the TTS system is trained such that the TTS system generates expressive speech.

Document <CIT> discloses a text-to-speech synthesis method based on a neural network architecture and having internal parameter sets linked to expressivity scores, like fundamental frequency profiles. The document also discusses enhancing the quality of the synthesised speech by training the method on datasets of audio books.

Systems and methods in accordance with non-limiting examples will now be described with reference to the accompanying figures in which:.

The present disclosure provides a text-to speech synthesis method as defined in claim <NUM>, a method of training a text-to-speech synthesis system as defined in claim <NUM>, a text-to-speech synthesis system as defined in claim <NUM> and a carrier medium comprising computer readable code as defined in claim <NUM>.

According to a first aspect, there is provided a text-to-speech synthesis method comprising:.

Methods in accordance with embodiment described herein provide an improvement to text-to-speech synthesis by providing a neural network that is trained to generate expressive speech. Expressive speech is speech that conveys emotional information and sounds natural, realistic and human-like. The disclosed method ensures that the trained neural network can accurately generate speech from text, the generated speech is comprehensible, and is more expressive than speech generated using a neural network trained using the first dataset directly.

In an embodiment, the expressivity score is obtained by extracting a first speech parameter for each audio sample; deriving a second speech parameter from the first speech parameter; comparing the value of the second parameter to the first speech parameter.

In an embodiment, the first speech parameter comprises the fundamental frequency.

In an embodiment, the second speech parameter comprises the average of the first speech parameter of all audio samples in the dataset.

In another embodiment, the first speech parameter comprises a mean of the square of the rate of change of the fundamental frequency.

In an embodiment, the second sub-dataset is obtained by pruning audio samples with lower expressivity scores from the first sub-dataset.

In an embodiment, audio samples with a higher expressivity score are selected from the first training dataset and allocated to the second sub-dataset, and audio samples with a lower expressive score are selected from the first training dataset and allocated to the first sub-dataset.

In an embodiment, the neural network is trained using the first sub-dataset for a first number of training steps, and then using the second sub-dataset for a second number of training steps.

In an embodiment, the neural network is trained using the first sub-dataset for a first time duration, and then using the second sub-dataset for a second time duration.

In an embodiment, the neural network is trained using the first sub-dataset until a training metric achieves a first predetermined threshold, and then further trained using the second sub-dataset. In an example, the training metric is a quantitative representation of how well the output of the trained neural network matches a corresponding audio data sample.

According to one example of the invention, there is provided a method of calculating an expressivity score of audio samples in a dataset, the method comprising: extracting a first speech parameter for each audio sample of the dataset; deriving a second speech parameter from the first speech parameter; and comparing the value of the second parameter to the first parameter.

The disclosed method provides an improvement in the evaluation of an expressivity score for an audio sample. The disclosed method is quick and accurate. Empirically, it has been observed that the disclosed method correlates well with subjective assessments of expressivity made by human operators. The disclosed method is quicker, more consistent, more accurate, and more reliable than assessments of expressivity made by human operators.

According to a second aspect, there is provided a method of training a text-to-speech synthesis system that comprises a prediction network, wherein the prediction network comprises a neural network, the method comprising:.

In an embodiment, the method further comprises training the neural network using a second training dataset. The neural network may be trained to gain further speech abilities.

In an embodiment the average expressivity score of the audio data in the second training dataset is higher than the average expressivity score of the audio data in the first training dataset.

According to a third aspect, there is provided a text-to-speech synthesis system comprising:.

wherein the first sub-dataset and the second sub-dataset comprise audio samples and corresponding text from the first training dataset and wherein the average expressivity score of the audio data in the second sub-dataset is higher than the average expressivity score of the audio data in the first sub-dataset.

In an embodiment, the system comprises a vocoder that is configured to convert the speech data into an output speech data. In an example, the output speech data comprises an audio waveform.

In an embodiment, the system comprises an expressivity scorer module configured to calculate an expressivity score for audio samples.

In an embodiment, the prediction network comprises a sequence-to-sequence model.

According to an example, there is provided speech data generated by a text-to-speech system according to the third aspect of the invention. The speech data disclosed is expressive and that conveys emotional information and sounds natural, realistic and human-likes.

In an embodiment, the speech data is an audio file of synthesised expressive speech.

According to a fourth aspect, there is provided a carrier medium comprising computer readable code configured to cause a computer to perform any of the methods above.

The methods are computer-implemented methods. Since some methods in accordance with examples can be implemented by software, some examples encompass computer code provided to a general purpose computer on any suitable carrier medium. The carrier medium can comprise any storage medium such as a floppy disk, a CD ROM, a magnetic device or a programmable memory device, or any transient medium such as any signa le.g. an electrical, optical or microwave signal. The carrier medium may comprise a non-transitory computer readable storage medium.

<FIG> shows a schematic illustration of a system <NUM> for generating speech <NUM> from text <NUM>. The system <NUM> can be trained to generate speech that is expressive. Expressive speech conveys emotional information and sounds natural, realistic and human-like. Quantitatively, the expressiveness of an audio sample is represented by an expressivity score; the expressivity score is described further below in relation to <FIG>, and <FIG>.

The system comprises a prediction network <NUM> configured to convert input text <NUM> into a speech data <NUM>. The speech data <NUM> is also referred to as the intermediate speech data <NUM>. The system further comprises a Vocoder that converts the intermediate speech data <NUM> into an output speech <NUM>. The prediction network <NUM> comprises a neural network (NN). The Vocoder also comprises a NN.

The prediction network <NUM> receives a text input <NUM> and is configured to convert the text input <NUM> into an intermediate speech data <NUM>. The intermediate speech data <NUM> comprises information from which an audio waveform may be derived. The intermediate speech data <NUM> may be highly compressed while retaining sufficient information to convey vocal expressiveness. The generation of the intermediate speech data <NUM> will be described further below in relation to Figure <NUM> (a).

The text input <NUM> may be in the form of a text file or any other suitable text form such as ASCII text string. The text may be in the form of single sentences or longer samples of text. A text front-end, which is not shown, converts the text sample into a sequence of individual characters (e.g. "a", "b", "c". In another example, the text front-end converts the text sample into a sequence of phonemes (/k/, /t/, /p/,.

The intermediate speech data <NUM> comprises data encoded in a form from which a speech sound waveform can be obtained. For example, the intermediate speech data may be a frequency domain representation of the synthesised speech. In a further example, the intermediate speech data is a spectrogram. A spectrogram may encode a magnitude of a complex number as a function of frequency and time. In a further example, the intermediate speech data <NUM> may be a mel spectrogram. A mel spectrogram is related to a speech sound waveform in the following manner: a short-term Fourier transform (STFT) is computed over a finite frame size, where the frame size may be <NUM>, and a suitable window function (e.g. a Hann window) may be used; and the magnitude of the STFT is converted to a mel scale by applying a non-linear transform to the frequency axis of the STFT, where the non-linear transform is, for example, a logarithmic function.

The Vocoder module takes the intermediate speech data <NUM> as input and is configured to convert the intermediate speech data <NUM> into a speech output <NUM>. The speech output <NUM> is an audio file of synthesised expressive speech and/or information that enables generation of expressive speech. The Vocoder module will be described further below.

In another example, which is not shown, the intermediate speech data <NUM> may be in a form from which an output speech <NUM> can be directly obtained. In such a system, the Vocoder <NUM> is optional.

<FIG> shows a schematic illustration of the prediction network <NUM> according to a non-limiting example. It will be understood that other types of prediction networks that comprise neural networks (NN) could also be used.

The prediction network <NUM> comprises an Encoder <NUM>, an attention network <NUM>, and decoder <NUM>. As shown in <FIG>, the prediction network maps a sequence of characters to intermediate speech data <NUM>. In an alternative example which is not shown, the prediction network maps a sequence of phonemes to intermediate speech data <NUM>. In an example, the prediction network is a sequence to sequence model. A sequence to sequence model maps a fixed length input from one domain to a fixed length output in a different domain, where the length of the input and output may differ.

The Encoder <NUM> takes as input the text input <NUM>. The encoder <NUM> comprises a character embedding module (not shown) which is configured to convert the text input <NUM>, which may be in the form words, sentences, paragraphs, or other forms, into a sequence of characters. Alternatively, the encoder may convert the text input into a sequence of phonemes. Each character from the sequence of characters may be represented by a learned <NUM>-dimensional character embedding. Characters from the sequence of characters are passed through a number of convolutional layers. The number of convolutional layers may be equal to three for example. The convolutional layers model longer term context in the character input sequence. The convolutional layers each contain <NUM> filters and each filter has a <NUM>×<NUM> shape so that each filer spans <NUM> characters. After the stack of three convolutional layers, the input characters are passed through batch normalization step (not shown) and ReLU activations (not shown). The encoder <NUM> is configured to convert the sequence of characters (or alternatively phonemes) into encoded features <NUM> which is then further processed by the attention network <NUM> and the decoder <NUM>.

The output of the convolutional layers is passed to a recurrent neural network (RNN). The RNN may be a long-short term memory (LSTM) neural network (NN). Other types of RNN may also be used. According to one example, the RNN may be a single bidirectional LSTM containing <NUM> units (<NUM> in each direction). The RNN is configured to generate encoded features <NUM>. The encoded features <NUM> output by the RNN may be a vector with a dimension k.

The Attention Network <NUM> is configured to summarize the full encoded features <NUM> output by the RNN and output a fixed-length context vector <NUM>. The fixed-length context vector <NUM> is used by the decoder <NUM> for each decoding step. The attention network <NUM> may take information (such as weights) from previous decoding steps (that is, from previous speech frames decoded by decoder) in order to output a fixed-length context vector <NUM>. The function of the attention network <NUM> may be understood to be to act as a mask that focusses on the important features of the encoded features <NUM> output by the encoder <NUM>. This allows the decoder <NUM>, to focus on different parts of the encoded features <NUM> output by the encoder <NUM> on every step. The output of the attention network <NUM>, the fixed-length context vector <NUM>, may have dimension m, where m may be less than k. According to a further example, the Attention network <NUM> is a location-based attention network.

According to one embodiment, the attention network <NUM> takes as input an encoded feature vector <NUM> denoted as h = {h1, h2,. A(i) is a vector of attention weights (called alignment). The vector A(i) is generated from a function attend(s(i-<NUM>), A(i-<NUM>), h), where s(i-<NUM>) is a previous decoding state and A(i-<NUM>) is a previous alignment. s(i-<NUM>) is <NUM> for the first iteration of first step. The attend() function is implemented by scoring each element in h separately and normalising the score. G(i) is computed from G(i) = Σk A(i,k)×hk. The output of the attention network <NUM> is generated as Y(i) = generate(s(i-<NUM>), G(i)), where generate() may be implemented using a recurrent layer of <NUM> gated recurrent units (GRU) units for example. The attention network <NUM> also computes a new state s(i) = recurrency(s(i-<NUM>), G(i), Y(i)), where recurrency() is implemented using LSTM.

The decoder <NUM> is an autoregressive RNN which decodes information one frame at a time. The information directed to the decoder <NUM> is be the fixed length context vector <NUM> from the attention network <NUM>. In another example, the information directed to the decoder <NUM> is the fixed length context vector <NUM> from the attention network <NUM> concatenated with a prediction of the decoder <NUM> from the previous step. In each decoding step, that is, for each frame being decoded, the decoder may use the results from previous frames as an input to decode the current frame. In an example, as shown in <FIG>, the decoder autoregressive RNN comprises two uni-directional LSTM layers with <NUM> units. The prediction from the previous time step is first passed through a small pre-net (not shown) containing <NUM> fully connected layers of <NUM> hidden ReLU units. The output of the pre-net, and the attention context vector are concatenated and then passed through the two uni-directional LSTM layers. The output of the LSTM layers is directed to a predictor <NUM> where it is concatenated with the fixed-length context vector <NUM> from the attention network <NUM> and projected trough a linear transform to predict a target mel spectrogram. The predicted mel spectrogram is further passed through a <NUM>-layer convolutional post-net which predicts a residual to add to the prediction to improve the overall reconstruction. Each post-net layer is comprised of <NUM> filters with shape <NUM> × <NUM> with batch normalization, followed by tanh activations on all but the final layer. The output of the predictor <NUM> is the speech data <NUM>.

The parameters of the encoder <NUM>, decoder <NUM>, predictor <NUM> and the attention weights of the attention network <NUM> are the trainable parameters of the prediction network <NUM>.

According to another example, the prediction network <NUM> comprises an architecture according to <NPL>.

Returning to <FIG>, the Vocoder <NUM> is configured to take the intermediate speech data <NUM> from the prediction network <NUM> as input, and generate an output speech <NUM>. In an example, the output of the prediction network <NUM>, the intermediate speech data <NUM>, is a mel spectrogram representing a prediction of the speech waveform.

According to an embodiment, the Vocoder <NUM> comprises a convolutional neural network (CNN). The input to the Vocoder <NUM> is a frame of the mel spectrogram provided by the prediction network <NUM> as described above in relation to <FIG>. The mel spectrogram <NUM> may be input directly into the Vocoder <NUM> where it is inputted into the CNN. The CNN of the Vocoder <NUM> is configured to provide a prediction of an output speech audio waveform <NUM>. The predicted output speech audio waveform <NUM> is conditioned on previous samples of the mel spectrogram <NUM>. The output speech audio waveform may have <NUM>-bit resolution. The output speech audio waveform may have a sampling frequency of <NUM>.

According to an alternative example, the Vocoder <NUM> comprises a convolutional neural network (CNN). The input to the Vocoder <NUM> is derived from a frame of the mel spectrogram provided by the prediction network <NUM> as described above in relation to <FIG>. The mel spectrogram <NUM> is converted to an intermediate speech audio waveform by performing an inverse STFT. Each sample of the speech audio waveform is directed into the Vocoder <NUM> where it is inputted into the CNN. The CNN of the Vocoder <NUM> is configured to provide a prediction of an output speech audio waveform <NUM>. The predicted output speech audio waveform <NUM> is conditioned on previous samples of the intermediate speech audio waveform. The output speech audio waveform may have <NUM>-bit resolution. The output speech audio waveform may have a sampling frequency of <NUM>.

According to another example, the Vocoder <NUM> comprises a WaveNet NN architecture such as that described in <NPL>.

According to a further example, the Vocoder <NUM> comprises a WaveGlow NN architecture such as that described in <NPL>.

According to an alternative example, the Vocoder <NUM> comprises any deep learning based speech model that converts an intermediate speech data <NUM> into output speech <NUM>.

According to another alternative embodiment, the Vocoder <NUM> is optional. Instead of a Vocoder, the prediction network <NUM> of the system <NUM> further comprises a conversion module (not shown) that converts intermediate speech data <NUM> into output speech <NUM>. The conversion module may use an algorithm rather than relying on a trained neural network. In an example, the Griffin-Lim algorithm is used. The Griffin-Lim algorithm takes the entire (magnitude) spectrogram from the intermediate speech data <NUM>, adds a randomly initialised phase to form a complex spectogram, and iteratively estimates the missing phase information by: repeatedly converting the complex spectrogram to a time domain signal, converting the time domain signal back to frequency domain using STFT to obtain both magnitude and phase, and updating the complex spectrogram by using the original magnitude values and the most recent calculated phase values. The last updated complex spectrogram is converted to a time domain signal using inverse STFT to provide output speech <NUM>.

<FIG> shows a schematic illustration of a configuration for training the prediction network <NUM> according to a comparative example. The prediction network <NUM> is trained independently of the Vocoder <NUM>. According to an example, the prediction network <NUM> is trained first and the Vocoder <NUM> is then trained independently on the outputs generated by the prediction network <NUM>.

According to an example, the prediction network <NUM> is trained from a first training dataset <NUM> of text data 41a and audio data 41b pairs as shown in <FIG>. The Audio data 41b comprises one or more audio samples. In this example, the training dataset <NUM> comprises audio samples from a single speaker. In an alternative example, the training set <NUM> comprises audio samples from different speakers. When the audio samples are from different speakers, the prediction network <NUM> comprises a speaker ID input (e.g. an integer or learned embedding), where the speaker ID inputs correspond to the audio samples from the different speakers. In the figure, solid lines (-) represent data from a training sample, and dash-dot-dot-dash (-··-) lines represent the update of the weights Θ of the neural network of the prediction network <NUM> after every training sample. Training text 41a in fed in to the prediction network <NUM> and a prediction of the intermediate speech data 25b is obtained. The corresponding audio data 41b is converted using a converter <NUM> into a form where it can be compared with the prediction of the intermediate speech data 25b in the comparator <NUM>. For example, when the intermediate speech data 25b is a mel spectrogram, the converter <NUM> performs a STFT and a non-linear transform that converts the mel spectrogram into audio waveform. The comparator <NUM> is compared the predicted first speech data 25b and the conversion of audio data 41b. According to an example, the comparator <NUM> may compute a loss metric such as a cross entropy loss given by: -(actual converted audio data) log (predicted first speech data). Alternatively, the comparator <NUM> may compute a loss metric such as a mean squared error. The gradients of the error with respect to the weights Θ of the RNN may be found using a back propagation through time algorithm. An optimiser function such as a gradient descent algorithm may then be used to learn revised weights Θ. Revised weights are then used to update (represented by -··- in <FIG> a and b) the NN model in the prediction network <NUM>.

The training of the Vocoder <NUM> according to an embodiment is illustrated in <FIG> and is described next. The Vocoder is trained from a training set of text and audio pairs <NUM> as shown in <FIG>. In the figure, solid lines (-) represent data from a training sample, and dash-dot-dot-dash (-··-) lines represent the update of the weights of the neural network. Training text 41a is fed in to the trained prediction network <NUM> which has been trained as described in relation to <FIG>. The trained prediction network <NUM> is configured in teacher-forcing mode - where the decoder <NUM> of the prediction network <NUM> is configured to receive a conversion of the actual training audio data 41b corresponding to a previous step, rather than the prediction of the intermediate speech data from the previous step - and is used to generate a teacher forced (TF) prediction of the first speech data 25c. The TF prediction of the intermediate speech data 25c is then provided as a training input to the Vocoder <NUM>. The NN of the vocoder <NUM> is then trained by comparing the predicted output speech 9b with the actual audio data 41b to generate an error metric. According to an example, the error may be the cross entropy loss given by: - (actual converted audio data 41b) log (predicted output speech 9b). The gradients of the error with respect to the weights of the CNN of the Vocoder <NUM> may be found using a back propagation algorithm. A gradient descent algorithm may then be used to learn revised weights. Revised weights Θ are then used to update (represented by -··- in <FIG>) the NN model in the vocoder.

The training of the Vocoder <NUM> according to another embodiment is illustrated in <FIG> and is described next. The training is similar to the method described for <FIG> except that training text 41a is not required for training. Training audio data 41b is converted into first speech data 25c using converter <NUM>. Converter <NUM> implements the inverse of the operation implemented by converter <NUM> described in relation to <FIG>. For example, when the intermediate speech data 25c is a mel spectrogram, the converter <NUM> performs the inverse of the non-linear transform that is performed by converter <NUM> and an inverse STFT. Thus, converter <NUM> converts the audio waveform into a mel spectrogram. The intermediate speech data 25c is then provided as a training input to the Vocoder <NUM> and the remainder of the training steps are described in relation to <FIG>.

<FIG> shows a schematic illustration of a configuration for training the prediction network <NUM> according to an embodiment. The audio data 41b from the first training dataset <NUM> is directed towards an Expressivity scorer module <NUM>. The Expressivity Scorer (ES) module <NUM> is configured to assign a score that represents the expressivity to each of the samples in audio 41b. The ES module <NUM> is described further below in relation to <FIG> and <FIG>. The score 41c corresponding to the audio data 41b is then directed into a Training Data Selector (TDS) module <NUM>. The TDS is configured to select text and audio data pairs from the training data set <NUM> according to the expressivity scores of the audio samples. The data selected by the TDS is referred to as the modified training dataset <NUM>. According to one example, the modified training dataset <NUM> is a dataset that comprises a copy of at least some of the audio and text samples from the first training dataset <NUM>. In another example, the modified training dataset <NUM> comprises a look up table that points to the relevant audio and text samples in the first training dataset <NUM>.

In an alternative embodiment which is not shown, the audio data 41b from the original training dataset <NUM> is assessed by a human operator. In this case, the human operator listens to the audio data 41b and assigns a score to each sample. In yet another alternative embodiment, the audio data 41b is scored by several human operators. Each human operator may assign a different score to the same sample. An average of the different human scores for each sample is taken and assigned to the sample. The outcome of human operator based scoring is that audio samples from the audio data 41b are assigned a score. As explained in relation to <FIG>, the score 41c corresponding to the audio data 41b is then directed into a Training Data Selector (TDS) module <NUM>.

In an embodiment, the audio data 41b is assigned a score by the human operator as well as a label indicating a further property. For example, the further property is an emotion (sad, angry, etc.. ), an accent (e.g. British English, French. ), style (e.g. shouting, whispering etc.. ), or non-verbal sounds (e.g. grunts, shouts, screams, um's, ah's, breaths, laughter, crying etc.. The TDS module is then configured to receive a label as an input and the TDS module is configured to select text and audio pairs that correspond to the inputted label.

In another embodiment, the label indicating the further property is assigned to the audio data 41b as it is generated. For example, as a voice actor records an audio sample, the voice actor also assigns a label indicating the further property, where, for example, the further property is an emotion (sad, angry, etc.. ), an accent (e.g. British English, French. ), style (e.g. shouting, whispering etc.. ), or non-verbal sounds (e.g. grunts, shouts, screams, um's, ah's, breaths, laughter, crying etc.. The TDS module is then configured to receive a label as an input and the TDS module is configured to select text and audio pairs that correspond to the inputted label.

According to another embodiment which described further below in relation to <FIG>, the TDS module <NUM> is further configured to select text and audio data pairs from a second dataset <NUM>. The second dataset <NUM> comprises text and audio data pairs that are not present in the first training dataset <NUM>. Optionally, the second dataset <NUM> further comprises: data from the same speaker; data from a different speaker; data from the same or a different speaker and conveying a new speech pattern such as emotion (e.g. sadness, anger, sarcasm, etc.. ), accents (e.g. British English, Australian English, French etc.. ), style (e.g. shouting, whispering etc.. ), or non-verbal sounds (e.g. grunts, shouts, screams, um's, ah's, breaths, laughter, crying etc..

The TDS module will be described further below in relation to <FIG>. In the example as shown in <FIG>, the modified training dataset <NUM> comprises a first sub-dataset <NUM>-<NUM>, a second sub-dataset <NUM>-<NUM>, or a third sub-dataset <NUM>-<NUM>; however, it will be understood that the modified training dataset <NUM> may generally comprise a plurality of sub-datasets, such as <NUM>, <NUM>, <NUM> and so on.

The method of training the prediction network <NUM> in the configuration shown in <FIG> will be described next. The training of the prediction network <NUM> differs from the training described in relation to <FIG> in that sub-datasets from the modified training dataset <NUM> are used instead of the first training data set <NUM>. When the modified training dataset <NUM> comprises more than one sub-dataset, the prediction network <NUM> may be trained in turn using each sub-dataset. The selection of sub-dataset is performed by the TDS module <NUM> and this is described further below in relation to <FIG>. For example, referring to the configuration of <FIG>, the prediction network <NUM> may initially be trained using first sub-dataset <NUM>-<NUM>, then with second sub-dataset <NUM>-<NUM>, and then with third sub-dataset <NUM>-<NUM>. The use of different sub-datasets may result in a prediction network <NUM> trained to generate intermediate speech <NUM> with high expressivity.

<FIG> shows a schematic illustration of the ES module <NUM> that takes the audio data 41b as input and generates score data 41c that corresponds to the audio data 41b.

<FIG> shows a schematic illustration of the determination of an expressivity score by the ES module <NUM> for different samples from the audio data 41b. In the example shown, the expressivity score is derived from a first speech parameter such as the fundamental frequency f<NUM> of the audio waveform. The fundamental frequency f<NUM> of the audio waveform may be estimated from the inverse of the glottal pulse duration, the glottal pulse duration being the duration between repeating patterns in the audio signal that are observed in human speech.

An example of an algorithm for estimating f<NUM> is the YIN algorithm in which: (i) the autocorrelation rt of a signal xt over a window W is found; (ii) a difference function (DF) is found from the difference between xt (assumed to be periodic with period T) and xt+T, where xt+T represents signal xt shifted by a candidate value of T; (iii) a cumulative mean normalised difference function (CMNDF) is derived from DF in (ii) to account for errors due to imperfect periodicities; (iv) an absolute threshold is applied to the value of the CMNDF to determine if the candidate value of T is acceptable; (v) considering each local minimum in the CMNDF; and (vi) determining which value of T gives the smallest CMNDF. However, it will be understood that other parameters such as the first three formants (F1, F2, F3) could also be used. It will also be understood that a plurality of speech parameters could be used in combination. The parameter f<NUM> is related to the perception of pitch by the human ear and is sometimes referred to as the pitch. In the examples shown in <FIG>, a plot of f<NUM> with time for each sample is shown. For the m = <NUM> sample, there are rapidly occurring peaks and troughs that occur with different spacings in the time domain waveform of the audio signal (second column) and this results in f<NUM> that varies significantly with time. Such a waveform generally represents an audio segment with high expressivity and might be attributed a maximum expressivity score of <NUM> for example. Conversely, in the sample m = <NUM>, the peak and troughs occur slowly and with about the same spacing and such a sample might be considered to have a low expressivity and might be attributed an expressivity score of <NUM>. The sample m = M shows an example with an intermediate expressivity score of <NUM>.

<FIG> shows a schematic illustration of the computation of an expressivity score performed by the ES module <NUM>. The audio data 41b is directed into the ES module <NUM>. In the initialisation step <NUM>, for each sample in the audio data 41b, the variation of fo(t) as a function of time is derived. For each sample m, the variation of fm<NUM>(t) is obtained and a time average <fm<NUM>(t)> is computed. The time average <fm<NUM>(t)> is the first speech parameter, for example. The value of fm<NUM>(t) is obtained using the Yin algorithm described above for example.

A second speech parameter is determined from the first speech parameter. According to an embodiment, the second speech parameter is obtained as the average of the first speech parameter <fm<NUM>(t)> for one or more samples in the dataset. In an embodiment, as shown in <FIG>, a discrete value for the expressivity score of an audio sample is computed by the ES module <NUM>.

According to another embodiment, the second speech parameter is obtained as the mean of the square of the rate of change of the fundamental frequency for one or more samples in the dataset. A discrete value for the expressivity score of an audio sample is computed by the ES module <NUM>.

According to another embodiment, a discrete value for the expressivity score of an audio sample is formed using emf and emv in combination.

According to an example, k = <NUM> such that discrete expressivity scores of <NUM>, <NUM>, <NUM>,. ,<NUM> are available. According to one example, a sample having an expressivity score of <NUM> or above is considered to be expressive. It will be understood, however, that samples having scores above any predetermined level may be considered to be expressive. For example, it may be preferred that a sample having a score above any value from <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>,<NUM>, <NUM>, <NUM> or any value therebetween, is considered to be expressive.

According to one example which is not shown, the average is the arithmetic mean, or median, or mode, of all the time averaged fm<NUM>(t). Furthermore, for each sample, the variability of fm<NUM>(t) for each sample, denoted as σm<NUM>, is computed. The average variability, which is the average value of σm<NUM> for all samples is determined. The average variability may be the arithmetic mean, or median, or mode of all values of σm<NUM>. The average variability is assigned an expressivity score of zero. For the other end of the scale, the maximum value of σm<NUM> over all m samples is identified and assigned a value of <NUM>. In step <NUM> and <NUM>, each sample is assigned an expressivity score equal to |σm<NUM> - average variability|×<NUM>. Although the example above describes a score in the range of <NUM> to <NUM>, it will be understood that the score could be in the range of <NUM> to <NUM>, or between any two numbers. Furthermore, it will be understood that although a linear scoring scale is described, other non-linear scales may also be used. The ES module <NUM> then outputs a score data 41c whose entries correspond to the entries of the audio data 41b.

In one embodiment, the expressivity score is computed for an entire audio sample, that is, for the full utterance.

In another embodiment, the expressivity score is computed for the audio sample on a frame-by-frame basis. The expressivity score computation is performed for several frames of the sample. An expressivity score for the sample is then derived from the expressivity scores for each frame, for example by averaging.

In another embodiment (which is not shown), the audio sample is further labelled with a further property. The further property label is assigned by a human operator for example. For example, the further property is an emotion (sad, happy, angry, etc.. ), an accent (e.g. British English, French. ), style (e.g. shouting, whispering etc.. ), or non-verbal sounds (e.g. grunts, shouts, screams, um's, ah's, breaths, laughter, crying etc.. In the calculation of the expressivity score described above in relation to <FIG>, the ES module <NUM> generates a score according to quantitative characteristics of the audio signal (as described above). Features, such as whether the audio sample conveys a particular emotion, accent, style or non-verbal sound, are not taken into consideration. Thus, an audio sample conveying sadness may have the same expressivity score as an audio sample conveying happiness, for example.

<FIG> is a schematic illustration of the selection of sub-datasets from the modified training dataset <NUM> by the TDS module according to an embodiment. According to this example, the sub-datasets <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> are selected such that the average expressivity score of the second sub-dataset <NUM>-<NUM> is greater than that of the first sub-dataset <NUM>-<NUM>, and that the average expressivity score of the third sub-dataset <NUM>-<NUM> is greater than that of the second sub-dataset <NUM>-<NUM>. Although <FIG> shows an example with three sub-datasets, any number of sub-datasets greater than two could be used as long as the average expressivity score of the sub-datasets is progressively higher. The effect of training the prediction network <NUM> with sub-datasets with increasing average expressivity scores is that the prediction network <NUM> is trained to generate highly expressive intermediate speech data. By training with a plurality of datasets with increasing average expressivity scores, the trained prediction network <NUM> generates highly expressive intermediate speech data <NUM>. By initially training with diverse samples having a low average expressivity score (e.g. sub-dataset <NUM>-<NUM>), the prediction network <NUM> learns to produce comprehensible speech from text accurately. This knowledge is slow to learn but is retained during training with subsequent sub-datasets containing expressive audio data. By progressively training with sub-datasets comprising samples having an increasing average expressivity score, the trained prediction network <NUM> learns to produce speech having a high expressivity. By contrast, if the prediction network <NUM> was not provided with increasingly expressive data sets for training, the prediction network <NUM> would learn to produce speech corresponding to the average of a diverse data set having a low average expressivity score.

The TDS module <NUM> is configured to change from one sub-dataset to another sub-dataset so that the prediction network <NUM> may be trained in turn with each sub-dataset.

In one embodiment, the TDS is configured to change sub-dataset after a certain number of training steps have been performed. The first sub-dataset <NUM>-<NUM> may be used for a first number of training steps. The second sub-dataset <NUM>-<NUM> may be used for a second number of training steps. The third sub-dataset <NUM>-<NUM> may be used for a third number of training steps. In one embodiment, the number of training steps are equal. In another embodiment, the number of training steps is different; for example the number of training steps decreases exponentially.

In another embodiment, the TDS is configured to change sub-dataset after an amount of training time has passed. The first sub-dataset <NUM>-<NUM> is used for a first time duration. The second sub-dataset <NUM>-<NUM> is used for a second time duration. The third sub-dataset <NUM>-<NUM> is used for a third time duration. In one embodiment, the time durations are equal. In another embodiment, the time durations are different, and, for example, are reduced when a sub-dataset is changed. For example, the first time duration is one day.

In another embodiment, the TDS is configured to change sub-dataset after a training metric of the neural network training reaches a predetermined threshold. In an example, the training metric is a parameter that indicates how well the output of the trained neural network matches the audio data used for training. An example of a training metric is the validation loss. For example, the TDS is configured to change sub-dataset after the validation loss falls below a certain level. In another embodiment, the training metric is the expressivity score as described in relation to <FIG>. In this case, the TDS is configured to change sub-dataset after the expressivity score of the intermediate speech 25b (which is converted to output speech <NUM> before scoring as necessary, for example, using converter <NUM>) generated by the prediction network <NUM> being trained reaches a predetermined threshold. In an example, when the expressivity scores are in the range of <NUM>, <NUM>,. <NUM>, a suitable threshold is <NUM>.

In yet another embodiment, the prediction network <NUM> is trained for a predetermined amount of time, and/or a number of training steps, and the performance of the prediction network <NUM> is verified on test sample text and audio pairs, and if the intermediate speech data <NUM> meets a predetermined quality, the sub-dataset is changed. In one embodiment, the quality is determined by a human tester who performs a listening test. In another embodiment, the quality is determined comparing the predicted intermediate speech data with the test audio data (which is converted using converter <NUM> if necessary) to generate an error metric. In yet another embodiment, the quality is determined by obtaining an expressivity score for the intermediate speech data 25b (which is converted to a time domain waveform if necessary) and comparing it with the expressivity score of the corresponding sample from the audio data 41b.

<FIG> shows a schematic illustration of the sub-datasets <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> and how they are obtained by pruning. In the example shown, sub-dataset <NUM>-<NUM> contains samples with a wide range of expressivity scores. Sub-dataset <NUM>-<NUM> may comprise all the samples of the first data set <NUM> for example. In sub-dataset <NUM>-<NUM>, the sample with a low expressivity score of <NUM> is pruned from sub-dataset <NUM>-<NUM>, thus increasing the average expressivity score of sub-dataset <NUM>-<NUM>. In sub-dataset <NUM>-<NUM>, the sample with a expressivity score of <NUM> is further pruned, thus further increasing the average expressivity score of sub-dataset <NUM>-<NUM>. In the example of <FIG>, a single sample is pruned from every sub-dataset; however, it will be understood that any number of samples may be pruned from every sub-dataset. In an example, the number of samples removed from each subsequent data set is equal to a pruning ratio×number of samples. In an example, the pruning ratio is <NUM>.

In another example, which is not shown, the sub-datasets <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> are obtained by sorting samples of the audio data 41b according to their expressivity scores, and allocating the lower scoring samples to sub-dataset <NUM>-<NUM>, the intermediate scoring samples to sub-dataset <NUM>-<NUM>, and the high scoring samples to sub-dataset <NUM>-<NUM>. When the prediction network <NUM> is trained using these sub-datasets in turn, the prediction network <NUM> may be trained to generate highly expressive intermediate speech data <NUM>.

<FIG> shows a schematic illustration of the selection of datasets for training the prediction network <NUM> by the TDS module <NUM> according to another embodiment. The TDS module <NUM> is configured to select data from the first training dataset <NUM> as well as another second training dataset <NUM>. According to one option, dataset <NUM> comprises audio data that is on average more expressive that the audio data of the first training dataset <NUM>. The second training dataset <NUM> comprises text 71a and corresponding audio 71b. Further training datasets (not shown) could be added. The second dataset <NUM> comprises samples that are not part of the first training dataset. According to an embodiment, the second dataset <NUM> further comprises: data from the same speaker; data from a different speaker; data from the same speaker and conveying a new speech pattern such as sadness, anger, sarcasm, etc.. ; or, data from a different speaker and conveying a new speech pattern such as sadness, anger, sarcasm, etc.. The TDS module <NUM> is configured to select data pairs from the first training dataset <NUM> to generate sub-dataset <NUM>-<NUM>, and to select data pairs from the second training dataset <NUM> to generate sub-dataset <NUM>-<NUM>. In one example, sub-datasets <NUM>-<NUM> and/or <NUM>-<NUM> are formed by using features such as expressivity scoring (as described in relation to <FIG>). In another example, sub-datasets <NUM>-<NUM> and/or <NUM>-<NUM> may be formed using a different selection procedure, such as selection by human operators. In a further example, sub-dataset <NUM>-<NUM> comprises all the samples of the first dataset <NUM>, and/or sub-dataset <NUM>-<NUM> comprises the samples of the second dataset <NUM>. Sub-dataset <NUM>-<NUM> is used to train the prediction network to generate output speech conveying a further property. For example, when sub-dataset <NUM>-<NUM> comprises speech patterns conveying emotions such as sadness, the prediction network <NUM> is trained to produce intermediate speech data <NUM> that sounds sad. In other examples, sub-dataset <NUM>-<NUM> comprises speech patterns reflecting a different gender, a different accent, or a different language. Therefore, the prediction network <NUM> can be trained to have additional abilities.

Although this is not shown, it will be understood that the example of <FIG> can be combined with features from <FIG> such as the Expressivity scorer.

In a further example which is not shown, the prediction network <NUM> can be trained initially to generate speech <NUM> according to any of the examples described in relation to <FIG>, <FIG>, <FIG>, <FIG>, <FIG> and/or 7b so that the generated speech is expressive. The prediction network <NUM> is further trained using the second dataset <NUM> in order to impart the model with a further ability. The initial training of the prediction network <NUM> can be understood as a pre-training step that gives the network the ability to generate expressive speech. This ability of the pre-trained prediction network <NUM> is used as a starting point and transferred during the further training with the second dataset <NUM> (transfer learning). The prediction network <NUM> that is pre-trained and then further trained according to this example retains expressive speech generation ability and gains a further ability.

<FIG> shows a schematic illustration of a text-to-speech (TTS) system according to an embodiment.

The TTS system <NUM> comprises a processor <NUM> and a computer program <NUM> stored in a non-volatile memory. The TTS system <NUM> takes as input a text input <NUM>. The text input <NUM> may be a text file and/or information in the form of text. The computer program <NUM> stored in the non-volatile memory can be accessed by the processor <NUM> so that the processor <NUM> executes the computer program <NUM>. The processor <NUM> may comprise logic circuitry that responds to and processes the computer program instructions. The TTS system <NUM> provides as output a speech output <NUM>. The speech output <NUM> may be an audio file of the synthesised speech and/or information that enables generation of speech.

The text input <NUM> may be obtained from an external storage medium, a communication network or from hardware such as a keyboard or other user input device (not shown). The output <NUM> may be provided to an external storage medium, a communication network, or to hardware such as a loudspeaker (not shown).

In an example, the TTS system <NUM> may be implemented on a cloud computing system, which transmits and receives data. Although a single processor <NUM> is shown in <FIG>, the system may comprise two or more remotely located processors configured to perform different parts of the processing and transmit data between them.

Claim 1:
A text-to-speech synthesis method comprising:
receiving text;
inputting the received text in a prediction network; and
generating speech data,
wherein the prediction network comprises a neural network, and wherein the neural network is trained by:
receiving a first training dataset comprising audio data and corresponding text data;
acquiring an expressivity score for each audio sample of the audio data, wherein the expressivity score is a quantitative representation of how well an audio sample conveys emotional information and sounds natural, realistic and human-like;
training the neural network using a first sub-dataset, and
further training the neural network using a second sub-dataset,
wherein the first sub-dataset and the second sub-dataset comprise audio samples and corresponding text from the first training dataset and wherein the average expressivity score of the audio data in the second sub-dataset is higher than the average expressivity score of the audio data in the first sub-dataset.