GUIDED SPEAKER ADAPTIVE SPEECH SYNTHESIS SYSTEM AND METHOD  AND COMPUTER PROGRAM PRODUCT

According to an exemplary embodiment of a guided speaker adaptive speech synthesis system, a speaker adaptive training module generates adaptation information and a speaker-adapted model based on inputted recording text and recording speech. A text to speech engine receives the recording text and the speaker-adapted model and outputs synthesized speech information. A performance assessment module receives the adaptation information and the synthesized speech information to generate assessment information. An adaptation recommendation module selects at least one subsequent recording text from at least one text source as a recommendation of a next adaption process, according to the adaptation information and the assessment information.

DETAILED DESCRIPTION OF THE EXEMPLARY EMBODIMENTS

The exemplary embodiment of a guided speaker adaptive synthesis technology makes a recommendation for a next adaptation by using such as the inputted recording speeches and the text contents, which guides the user inputting speech data again to perform reinforcing for the deficiencies of a previous adaptation process. Wherein the performance assessment may be divided into a coverage assessment and a spectral distortion assessment. In the exemplary embodiments, the estimation result of the coverage rate and the spectral distortion may be coupled with an algorithm, such as the design of a greedy algorithm, which selects the most suitable adaptation sentence from a text source and returns the assessment results to the user or the client, or to a module to handle the text and speech input. Wherein the coverage rate may be obtained by converting the input text to a string of a readable full label format, and then analyzing the coverage rate corresponding to the phone and the speaker-independent model content. The spectral distortion may be determined by comparing the spectral parameters of both the recording speech and the adapted synthesized speech after time alignment.

Speaker adaptation basically uses the adaptation data to adapt all speech models. The speech models may be multiple HMM spectrum models, multiple HMM duration models and the multiple HMM pitch models, which are referred by a HMM-based framework. In the exemplary embodiments, the adapted speech models in the speaker adaptation process may be, but not limited to the HMM spectrum models, the HMM duration models and the HMM pitch models, which are referred by a HMM-based framework. Takes the aforementioned HMM-based models as an example for illustrating the speaker adaptation and training. Theoretically, if the model numbers mapped to the adaptation data with full label format are widely distributed, that is, the adaptation data can be used to adapt most of models in the original TTS system, the obtained adaptation results should be better. Based on this viewpoint, the exemplary embodiments design a selection method, such as a greedy algorithm, to maximize the model coverage, and the selection method selects at least one subsequent recording text to perform the speaker adaptation efficiently.

The state of the art speaker adaptation technique performs the adaptation training of speech independent (SI) speech synthesis models according to inputted recording speech to generate a speech adaptive (SA) speech synthesis models, and uses a TTS engine to perform directly the speech synthesis according to the SA speech synthesis models. Different from the current technologies, the exemplary embodiments of the speech synthesis system in the disclosure, a performance assessment module and an adaptation recommendation module are added to make a recommendation of different subsequent recording texts according to the current results in the speaker adaptation process, and provide users (clients) with assessment information of current adaptation speech for reference. The performance assessment module may estimate the phone coverage, the model coverage and the spectral distortion for the adaptation speech. The adaptation recommendation module may select at least one subsequent recording text from the text source as a recommendation of the next adaptation, according to the adaptation results. The assessment information of the current adaptation speech is estimated by the performance assessment module. Accordingly, by the way of constant adaptation and providing the text recommendation for performing effective speaker adaptation, the speech synthesis system may provide the good sound quality and similarity.

Accordingly,FIG. 4shows a guided speaker adaptive speech synthesis system, according to an exemplary embodiment. Refer toFIG. 4, a speech synthesis system400comprises a speaker adaptive training module410, a text-to-speech (TTS) engine440, a performance assessment module420, and an adaptation recommendation module430. The speaker adaptive training module410adapts a speaker adaptive model416according to a recording text411and at least one recording speech412. The speaker adaptive training module410performs an analysis according to the recording text411, collects corresponding phone information and model information of the recording text411. An adaptation information414produced by the speaker adaptive training module410includes at least the inputted recording speech412, phonetic information generated by analyzing the recording speech412, corresponding phone and a variety of model information of the recording text411. The variety of model information used may be such as spectrum model information and prosody model information. The prosody model is the pitch model, since spectrum determines timbre and pitch has a key influence on the speech prosody.

A text-to-speech (TTS) engine440outputs synthesized speech information442according to the recording text411and the speaker adapted model416. The synthesized speech information442includes at least the synthesis speech and the voiced segment information of the synthesized speech.

The performance assessment module420combines with the adaptation information414and the synthesized speech information442to estimate the assessment information of a current adapted speech. The assessment information comprises such as phone and model coverage rate424, and one or more speech distortion assessment parameters (for example, spectral distortion422, etc.). The phone and model coverage rate424includes, such as phone coverage rate, spectrum model coverage rate, pitch model coverage rate. Once statistical information of phone and model is obtained, the phone and model coverage rate may be calculated by applying the phone coverage formula and the model coverage formula. The estimation of the one or more speech distortion assessment parameters (such as spectral distortion and/or pitch distortion, etc.) may be obtained by using the inputted recording speech of the speaker adaptive training module410and the speech segment information of the recording speech and the synthesized speech provided by the TTS engine440, and through a plurality of performing procedures. The detailed about how to estimate phone and model coverage rate and speech distortion assessment parameters will be described as follows.

The adaptation recommendation module430selects at least one subsequent recording text from a text source (for example, a text database)450, as the recommendation of the next adaptation, according to the adaptation information414outputted from the speaker adaptation training module410and the assessment information of a current recording speech, such as spectral distortion, estimated by performance assessment module420. The strategy of selecting the recording text by the adaptation recommendation module430, may be such as, maximizing the phone/model coverage rate. The speech synthesis system400may output the assessment information of the current adapted speech estimated by the performance assessment module420, such as the phone and model coverage rate, spectral distortion, etc., and the recommendation for the next adaptive speech made by the adaptation recommendation module430, such as the recommendation of recording text, to an adaptation result output module460. The adaptation result output module460may send back this information, such as the assessment information, recording text's recommendation, to the user or the client, or to a text and speech input processing module. Thus, the efficient speaker adaptation may be performed through constant adaptation and providing text recommendation, which makes the speech synthesis system400able to output the adapted synthesized voice with better quality and higher similarity via the adaptation results output module460.

FIG. 5shows an example illustrating the speaker adaptive training module which collects the corresponding phone and the model information for each of full label information from an input text, according to an exemplary embodiment. In the example ofFIG. 5, the speaker adaptive training module converts the input text into multiple pieces of full label information516, compares the multiple pieces of full label information516and collects corresponding phone information of each of the multiple pieces of full label information, the cepstral model numbers of states 1 to 5, and pitch model numbers of states 1-5. The more the model types are collected (higher coverage), the better the speech-adapted model may be obtained.

It may be seen from theFIG. 5, when a piece of full label information is inputted to a speech synthesis system, the cepstral model numbers and the pitch model numbers may be obtained by using such as a decision tree. The phone information of the piece of full label information may also be seen from the full label information itself Take the full label information, i.e. sil-P14+P41/A:4̂0/B:0+4/C:1=14/D:1@6, as an example, the phone is P14 (corresponding to a phonetic alphabet), while the left phone is sil (represents silence), and the right phone is P41 (corresponding to another phonetic alphabet). Thus collecting phone and model information of adapted speech data is quite intuitive, and this information-gathering process is executed in the adaptive training module. Once the statistical information of phones and models is obtained, one may apply the formula for the phone coverage rate and formula for the model coverage rate to estimate the phone and model coverage rate.

FIG. 6shows an exemplar for estimating the phone coverage rate and the model coverage rate, according to an exemplary embodiment. In the coverage rate calculation formula610ofFIG. 6, the value of the denominator (50 in this case) in the formula for estimating phone coverage rate represents 50 different phones in the TTS engine; and assuming that each of the cepstral model and the pitch model has five different states in the formulas for estimating model coverage. When the model is the cepstral model, the denominator of StateCoverRates(i.e., variable ModelCounts) represents the overall type number of the cepstral model of the state s, and the nominator (i.e., variable Num_UniqueModels) represents the type number of the current collected cepstral model of the state s. According to the formula for estimating the model coverage rate, one may estimate the cepstral model coverage rate. Similarly, when the model is the pitch model, one may estimate the pitch model coverage rate from the formula for estimating the model coverage rate.

When the speech distortion assessment parameters estimated by the performance assessment module420contains the spectral distortion, it is more complex compared with the coverage rate calculation.FIG. 7shows the operation for estimating the spectral distortion by the performance assessment module, according to an exemplary embodiment. As shown inFIG. 7, the spectral distortion estimation may be obtained by using the recording speech outputted from the speaker adaptive training module410, the voiced segment information of the recording speech, and the synthesized speech and its voiced segment information provided by the TTS engine440, and by further performing a feature extraction710, a time alignment720, and a spectral distortion calculation730.

The feature extraction is firstly calculating the parameters of the speech, such as using the Mel-Cepstral parameter, or the linear prediction coding (LPC), or the line spectrum frequency (LSF), or the perceptual linear prediction (PLP) etc., as the reference speech features, then performing the time alignment comparing of the recording speech and the synthesized speech. Although voiced segment information of the recording speech and the synthesized speech are both known, the pronunciation duration of each word of the two kinds of speech is not identical. Thus time alignment is needed before calculating the spectral distortion. The Dynamic Time Warping (DTW) technique may be used for time alignment. Finally such as the Mel-Cepstral distortion (MCD) is taken as basis for calculating the spectral distortion indicator. The calculation formula of the MCD is as follows:

wherein mcp is the Mel-Cepstral parameter, syn comes from the synthesized frame of the adapted speech, tar comes from the target frame of the real speech, and N is the mcp dimension. The spectral distortion of each speech unit (such as phone) may be estimated as follows:

K is the number of the frames.
When the MCD value becomes higher, it represents that the similarity of the synthesis result is lower. Therefore, the current adaptation result of the system may be represented by this indicator.

The adaptation recommendation module430combines the adaptation information414from the adaptive training module410, and the assessment information estimated from the performance assessment module420such as the spectral distortion, to select a recommendation of at least one subsequent recording text from a text source.FIG. 8illustrates a schematic diagram for the operation of the adaptation recommendation module, according to an exemplary embodiment.FIG. 8shows the operation of the adaptation recommendation module, according to an exemplary embodiment. As shown inFIG. 8, the adaptation recommendation module430further utilizes a phone/model based coverage maximization algorithm820, such as the greedy algorithm, to select the most suitable recording text, and in the process of executing this algorithm, refers to the result of a weight re-estimation810; and then outputs the recommendation of the subsequent recording text.

According to the above description on the guided speaker adaptive speech synthesis system and each component thereof,FIG. 9shows a guided speaker adaptive speech synthesis method, according to an exemplary embodiment. As shown inFIG. 9, this speech synthesis method900firstly inputs the recording text and the corresponding recording speech to perform speaker adaptation training, and outputs a speaker-adapted model and adaptation information (step910). Then it provides the speaker-adapted model and the recording text to a TTS engine, and outputs synthesized speech information (step920). This speech synthesis method900further estimates assessment information of a current recording speech, according to the adaptation information and the synthesis speech information (step930). Finally according to the adaptation information and the assessment information, the speech synthesis method900selects at least one subsequent recording text from a text source as the recommendation of a next adaptation (step940).

Accordingly, the guided speaker adaptive speech synthesis method may comprise: inputting at least one recording text and at least one recording speech, and outputting an adaptation information and a speaker adaptive model; loading the speaker adaptive model and a given recording text, and outputting a synthesized speech information; inputting the adaptation information and the synthesized speech information, and estimating an assessment information; and selecting one or more subsequent recording texts from at least one text source as a recommendation of a next adaption process, according to the adaptation information and the assessment information.

The adaptation information includes at least the recording speech, and voiced segment information of the recording speech and the corresponding phone and model information of the recording speech. The synthesized speech information includes at least the synthesized speech and its voiced segment information. The assessment information includes at least phone and model coverage rate, and one or more speech distortion assessment parameters (such as the spectral distortion).

In the speech synthesis method900, related contents on how to collect the corresponding phone and model information from the recording speech of an input text, and how to estimate the phone coverage rate and the model coverage rate, how to estimate the spectral distortion and the strategy of selecting the recording text have been described in the foregoing exemplary embodiments, and is not restated here. As stated before, the exemplary embodiments of the present disclosure firstly performs a weight re-estimation; then uses the phone and model based coverage maximization algorithms to select the recording text.FIG. 10andFIG. 11illustrate flow charts for a phone based coverage maximization algorithm and a model based convergence maximization algorithm, respectively, according to exemplary embodiments.

Refers to the flow chart inFIG. 10, firstly the phone based coverage maximization algorithm performs a weight re-estimation according to a current assessment information (step1005). A new weight (PhoneID) of a phone and an updated influence (PhoneID) of the phone are obtained after the weight re-estimation is performed, wherein PhoneID is an identifier of the phone. The details of this weight re-estimation will be described inFIG. 12. Then, the score of each candidate sentence of a text source is initialized as 0 (step1010); the algorithm is based on the definition of a score function to calculate the score of each sentence in the text source, and normalizes the score (step1012); such as according to the number of phones in the sentence to perform this normalization (e.g., divide the total score by the number of phones). An exemplar for defining the score function of a phone is as follows:

In the score function mentioned above, the score of a phone is determined by the weight and the influence of the phone. The weight(PhoneID) value is the reciprocal of the number of occurrences for PhoneID in a large text corpus. In other words, the higher the number of occurrences is, the lower the weight(PhoneID) value is. The impact(PhoneID) value is initialized to some natural number, e.g. 20, and will be decreased by one (till zero) whenever PhoneID is picked up during the selection process. Such design can reflect their lessening importance in the next iteration.

The more the phone categories, the higher the candidate sentence's score. Finally at least one candidate sentence with highest score is selected and removed from the text source to a sentence set of the adaptation recommendation (step1014), and the influence of the phones contained in the selected sentence is reduced (step1016), in order to facilitate the next selecting opportunity of other phones. When the number of the selected sentences does not exceed a predetermined value (step1018), then step1012is performed, and the scores of all remaining candidate sentences in the text source are re-calculated. And the above process is repeated until the number of selected sentences exceeds the predetermined value.

Refers to the flowchart inFIG. 11, first of all, this model based coverage maximization algorithm performs a weight re-estimation according to a current assessment information (step1105). After the weight re-estimation is performed, a new MCP weight and a new LF0 weight of these two models, and two update influences Influence(MsL) and Influence(PsL) of these two models may be obtained, wherein MsLindicates a corresponding spectral (MCP) model when the state is s and the text label information is L. Similarly, PsLindicates a corresponding pitch (LF0) model when the state is s and the text label information is L. The text label information is defined as the full label information obtained after the inputted recording text and through a text analysis of the speaker adaptive training module, as shown in516ofFIG. 5. The details of the weight re-estimation will be described in theFIG. 12. Then, the exemplary embodiment initializes the score for each candidate sentence in a text source to 0 (step1110). This algorithm is based on the definition of a score function to calculate the score of each sentence in the text source, and normalizes the score (step1112), such as by performing the normalization (e.g., divide the total score by the number of phones) based on the number L (text label) in the sentence. An exemplary embodiment for defining score function of a model is as follows:

In the score function mentioned above, the score is determined according to a cepstral model score and a pitch model score. A cepstral model score or a pitch model score is determined by the weight and the influence of the model. In the model score function mentioned above, the system initializes the cepstral model's weight Weight(MsL) and the pitch model's weight Weight(PsL), by taking the reciprocal of the number of occurrences as the MCP models and the LF0 models. Therefore, the more frequently the model appears in the storage medium e.g. the data corpus, the lower its model weight is. The values of Influence(MsL) and Influence(PsL) are initialized to a natural number, for example, five. The value is decreased by one whenever MsLis picked up during the selection process. Such design can reflect their lessening importance in the next iteration.

The candidate sentence with more MCP and LF0 models types may obtain a higher score. Finally, at least one candidate sentence with a highest score is selected and removed from the text source to a sentence set of the adaptation recommendation (step1114), and the influence of the models contained in the selected sentence is reduced (step1116), in order to facilitate the next selecting opportunity of other models. When the number of the selected sentences does not exceed a predetermined value (step1118), then step1112is performed. And, the scores of all remaining candidate sentences in the text source are re-calculated, and the above process is repeated until the number of selected sentences exceeds the predetermined value.

In other words, model based coverage maximization algorithm defines a score function of a model to perform the score estimation for each candidate sentence in a text source. The more model types a candidate sentence has, the higher its score will be. Finally, at least one candidate sentence with the highest score is selected and removed from the text source into a sentence set of the adaptation recommendation, and the influence of the models contained in the selected sentence is reduced in order to facilitate the next selecting opportunity of other models. Then the scores of all remaining candidate sentences in the text source are re-calculated, and the above process is repeated until the number of selected sentences exceeds the predetermined value.

According to the aforementioned in the flow charts ofFIG. 10andFIG. 11, in the phone based coverage maximization algorithm or the model based coverage maximization algorithm, the weight re-estimation plays a key role. It determines, based on the spectral distortion, the new phone weight and the new model weight such as new Weight(PhoneID), Weight (MsL) and Weight(PsL), and uses a timbre similarity method to dynamically adjust the level of the weight. The weight re-estimation uses the timbre similarity method to dynamically adjust the level of the weight, so that the reference for selecting at least one subsequent text not only takes the coverage (based only on the text reference) into account but also considers the feedback of the synthesis result. The timbre similarity usually based on the spectral distortion to estimate. If the spectral distortion of a speech unit (such as phone or syllable or word) is too high, it indicates that the adaptation result is not good enough and the subsequent text should strengthen the selection of this unit, therefore its weight should be increased. On the contrary, when the spectral distortion of a speech unit is very low, it indicates that the adaptation result has been good enough, and the weight of the subsequent text should be lowered, so that the selecting opportunities of other speech units may be increased. Thus, in the disclosed exemplary embodiments, the weight adjustment principle is, when the spectral distortion of a speech unit is higher than a high threshold value (e.g., the mean distortion of the original speech plus the standard deviation of the original speech), increases the weight of the speech unit; when the spectral distortion of a speech unit is lower than a low threshold value (e.g., the mean distortion of the original speech minus the standard deviation of original speech), decreases the weight of the speech unit.

FIG. 12shows an adjustment scheme of the weight re-estimation, according to an exemplary embodiment. In the formula1200of the adjustment scheme of the weight re-estimation shown inFIG. 12, Di represents the i-th distortion of a speech unit (e.g., phone unit), Dmeanrepresents the mean distortion of the adaptation data, Dstdrepresents the standard deviation of the adaptation data. N represents the number of units involved in this weight adjustment (for example, five involved in the calculation of phone P14). Each factor, Factori, estimated by the same speech unit is not identical, therefore the mean of these factors (i.e., the mean factor F) is taken as the representative. Finally, adjusting the new weight is performed according to the mean factor F. One exemplary adjustment formula is new weight=weight×(1+F), wherein the value of the mean factor F may be a positive value or a negative value.

FIG. 13illustrates a schematic view illustrating the spectral distortion between the synthesized speech and the original for a sentence of which the spectral distortion calculation unit is a phoneme, according to an exemplary embodiment, wherein the horizontal axis represents different phones, the vertical axis represents the spectral distortion (the unit of the vertical axis is dB), and the speech unit for calculating the spectral distortion is phone. Since the spectral distortions of phones 5 to 8 are higher than (Dmean+Dstd), therefore, according to the weight adjustment principle of the disclosed exemplary embodiments, weights of phone 5, phone 6, phone 7 and phone 8 are increased; while the spectral distortions of phone 11, phone 13, phone 20 and phone 37 are lower than (Dmean−Dstd), therefore, according to the weight adjustment principle of the disclosed exemplary embodiments, weights of phone 11, phone 13, phone 20, and phone 37 are decreased.

In the above exemplary embodiment of the guided speaker adaptive speech synthesis method may be implemented by a computer program product. The computer program products may use at least one hardware processor to read program codes embedded in a storage media to execute this method. Yet in accordance with one exemplary embodiment of the disclosure, the computer program product may comprise a storage media having a plurality of readable program codes, and use the at least one hardware processor reading the readable program code embedded in the storage media to execute: inputting at least one recording text and at least one recording speech, and outputting an adaptation information and a speaker adaptive model; loading the speaker adaptive model and a given recording text, and outputting a synthesized speech information; inputting the adaptation information and the synthesized speech information, and estimating an assessment information; and selecting one or more subsequent recording texts from at least one text source as a recommendation of a next adaption process, according to the adaptation information and the assessment information.

In summary, the disclosed exemplary embodiments provide a guided speaker adaptive speech synthesis system and method. Its technology inputs at least one recording text and at least one recording speech, and outputs adaptation information and a speaker adaptive model; a TTS engine reads the speaker adaptive model and the recording text, and outputs synthesized speech information; then combines with the adaptation information and the synthesized speech information, and estimates assessment information, and selects at least one subsequent recording text according to the adaptation information and the assessment information as a recommendation for a next adaptation. This technique considers phone and model coverage rate, selects speech with the distortion as the criteria, and makes a recommendation for a next speech adaption, thereby guiding users/clients reinforcing input the speech data corpus for the deficiencies of a previous adaptation process, to provide good voice quality and similarity.