Patent Publication Number: US-7590540-B2

Title: Method and system for statistic-based distance definition in text-to-speech conversion

Description:
FIELD OF THE INVENTION 
     This invention relates to text-to-speech conversion (TTS). More particularly, this invention relates to a method and system for statistics-based distance definition in text-to-speech conversion. 
     BACKGROUND OF THE INVENTION 
     Text-to-speech conversion refers to the technology that intelligently converts words into natural voice flow by using the designs of advanced natural language processing algorithms under the support of computers. TTS facilitates user interaction with the computer, thereby improving the flexibility of the application system. 
     A typical TTS system as shown in  FIG. 1  comprises a text analysis unit  101 , a prosody prediction unit  102  and a speech synthesis unit  103 . The text analysis unit  101  is responsible for parsing the input plain text into rich text with descriptive prosody annotations such as pronunciations, stresses, phrase boundaries and pauses. The prosody prediction unit  102  is responsible for predicting the phonetic representation of prosody, such as values of pitch, duration and energy of each synthesis segment, according to the result of text analysis. The speech synthesis unit  103  is responsible for generating intelligible voices as a physical result of the representation of semantics and prosody information implicitly contained in the plain text. 
     For example, performing TTS on the text   will result in the following. First the text is input into the text analysis unit  101 , so that the pronunciation of each character and the phrase boundaries are identified as follows. The following example uses Chinese language text, but of course the present invention may be applied to any language.
           .   zhe4 shi4 yi2 ge4 zhuan1 li4 shen1 qing3       

     With the above text analysis, the prosody prediction unit  102  performs prosody prediction on the characters in the text. Then, the speech synthesis unit  103  will produce the voice corresponding to said text based on the predicted prosody information. In current TTS technologies, statistics-based distance definition approaches are an important tendency. In these kinds of approaches, text analysis and prosody prediction models are trained from a large labeled corpus, and speech synthesis is always based on selection of multiple candidates for each synthesis segment. A general framework for the TTS-based corpus is shown in  FIG. 2 . 
     In statistics based approaches, especially in prosody prediction and inventory based selection, many difficult problems involve the distance definition between a sample and a given cluster. Even with complex contexts to cluster data, the problem of data dispersing is so serious in almost every cluster, and the overlap among clusters is so serious, that it is difficult to evaluate whether the sample belongs to the given cluster. 
     There are some classical definitions used in current TTS, such as the weighted Euclid distance and the Mahalanobis distance. For the Euclid distance, by using an average of the used sample points as the sample point, it is often difficult to choose the most appropriate value to be the sample point. Moreover, the relationship among different dimensions may be ignored or poorly modeled by pre-given knowledge. A problem with the Mahalanobis distance is the poor capability to simulate the complex distribution. 
       FIG. 3  is a histogram, with the duration distribution of a sample in a cluster in a TTS corpus being a log distribution. As shown in  FIG. 3 , the data is so dispersive that the mean value approach of the Euclid distance is not able to simulate its distribution, and Mahalanobis distance seems difficult for a refined simulation also because it is not a normal distribution. 
     SUMMARY OF THE INVENTION 
     In consideration of the above problems, the present invention is proposed, where the Gaussian Mixture Model (GMM) is applied to distance definition in TTS. More particularly, the invention relates to a novel statistics-based distance definition approach used for text-to-speech conversion. In the distance definition according to the present invention, probability distribution is prominently adopted through the GMM. The present invention may be used to better solve such difficulties as data sparseness and data dispersing in TTS statistical technology by using of the probability distribution, as compared with the afore-mentioned Euclid distance and Mahalanobis distance. GMM is an algorithm to describe some complex distribution by a cluster of Gaussian models with simple parameters for each Gaussian model. For example, the distribution of  FIG. 3  can be simulated by a GMM combined with two Gaussian models.  FIG. 4  is the illustration of the simulation. Although for illustrative a distribution is shown in  FIG. 3  using two Gaussian distributions, it will be understood by those skilled in the art that it is possible to use more than two distributions as required. 
     According to embodiments of the invention, there is provided a method for distance definition in the TTS system, comprising the steps of: analyzing the text that is to be subjected to TTS, to obtain a text with descriptive prosody annotation; performing clustering for the samples in the obtained text; and generating a GMM model for each cluster, to determine the distance between the sample and the corresponding GMM model. According to embodiments of the invention, there is provided a system for distance definition in the TTS system, comprising: a text analysis unit, for analyzing the text that is to be subjected to TTS, to obtain a text with descriptive prosody annotation; a prosody prediction unit, for performing clustering for the samples in the text obtained by the text analysis unit; and a GMM model base, connected to said prosody prediction unit, for storing the generated GMM models. These first and second aspects of the invention are directed to training the GMM models by using the corpus. 
     According to embodiments of the invention, there is provided a method for speech synthesizing in the TTS system, comprising the steps of: determining the cluster for the unit to be subjected to TTS, thereby to determine the GMM model of said cluster; calculating the distance between the candidate samples in the cluster and the determined GMM model; and identifying the sample with the smallest distance for subsequent speech synthesizing. According to embodiments of the invention, there is provided a system for speech synthesizing in the TTS system, comprising: a cluster determining unit, for determining the cluster for the unit to be subjected to TTS, thereby to determine the GMM model of said cluster; a distance calculating unit, for calculating the distance between the candidate samples in the cluster and the determined GMM model; and an optimizing unit, for identifying the sample with the smallest distance for subsequent speech synthesizing. These third and forth aspects of the invention are directed to speech synthesis by using GMM models. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a block diagram of a typical TTS system; 
         FIG. 2  is a block diagram of a general corpus-based TTS; 
         FIG. 3  shows a log duration distribution of a sample in a cluster of a TTS corpus; 
         FIG. 4  is a diagram showing the simulation of the distribution of  FIG. 3  by using GGM combined with two Gaussian models; 
         FIG. 5  is a flowchart for the training process of the method according to embodiments of the present invention; 
         FIG. 6  is a diagram of the decision tree used for clustering the samples; 
         FIG. 7  is a block diagram for the training section of the system according to embodiments of the present invention; 
         FIG. 8  is a flowchart for the synthesizing process of the method according to embodiments of the present invention; 
         FIG. 9  is a diagram for dynamic planning according to embodiments of the invention; 
         FIG. 10  is a block diagram for the synthesizing section of the system according to embodiments of the present invention; 
         FIGS. 11 and 12  are block diagrams for the cluster determining unit, distance calculating unit and the optimizing unit; 
         FIG. 13  shows all the data in a leaf in the pitch tree; and 
         FIG. 14  shows a situation where there are unreasonable jumps between neighboring units. 
     
    
    
     DETAILED DESCRIPTION 
     Embodiments of the invention will be described in connection with the drawings. However, it should be readily understood that these embodiments are illustrative only and should not be taken as limiting the scope of the invention. 
     A GMM portrays the distribution of the samples in the current cluster. For a position where the distribution is dense, the output probability is large, and for a position where the distribution is sparse, the output probability is small. The distance between a unit and a GMM model describes the degree of approximation between the unit and the cluster where the model is located. With GMM being an abstract representation of said cluster, the distance between a unit and the GMM model can be depicted by using the probability output of the unit in that model, the larger the probability, the smaller the distance, and vice versa. 
     Assuming that G represents the GMM model, the probability output of unit X in G is P(X|G), and the distance definition between X and G is D(X, G). Where there are two units X 1  and X 2 , if P(X 1 |G)&gt;P(X 2 |G), then D(X 1 , G)&lt;D(X 2 , G); if P(X 1 |G)&lt;P(X 2 |G), then D(X 1 , G)&gt;D(X 2 , G); and if P(X 1 |G)=P(X 2 |G), then D(X 1 , G)=D(X 2 , G). 
     Now, reference is made to  FIG. 5 , where the flowchart for the training stage for the method according to embodiments of the invention is shown. The method starts from step S 510 , and then proceeds to step S 520 . Step S 520  is to analyze the text to be TTS converted, so as to attain text with descriptive prosody annotation. Then, the method proceeds to step S 530 , where the samples in the text is clustered. As is known by a skilled person, the “sample” can mean the condition on which the modeling is based, for example, if the duration is to be modeled, then the duration itself is a sample. After the samples are clustered, the method proceeds to step S 540 , where a GMM model is generated for each cluster. With the generation of the GMM model, the method ends with steps S 550 . The generated GMM model will be used in the subsequent speech synthesis process, as is described later. 
     Next, the specific way for clustering the samples will be elaborated. As is known by those skilled in the art, the samples can be clustered in numerous ways. For example, the samples can be clustered by dimensions, or by such conditions as “duration”. However, according to embodiments of the invention, the samples are clustered by using the decision tree. The decision tree is a data-driven auto-clustering method, wherein the clustering is decided through data, whereby it is unnecessary for the user to be knowledgeable about clustering. In TTS, decision tree is popularly used for context dependent clustering or prediction. There can be various types of decision trees, and  FIG. 6  shows the main idea of a decision tree. 
     All of the data in the parent node of the tree is split into two child nodes by an optimized question from a pre-defined question set. Following a pre-defined criteria, the distance in any child node is small and between two child nodes is large. After each split process, an optional function can be done to merge the similar nodes among all of the leaves. All of the splitting, stop-splitting and merging are optimized by the pre-defined criteria. 
     Reference is now made to  FIG. 6 , assuming that condition  1  is if at the beginning of the sentence, condition  2  is if at the forth tone, and condition  3  is if a light tone is followed. If a sample traverses enough nodes in the decision tree (here, 3 nodes are shown for the purpose of illustration) for achieving a suitable cluster, a GMM model is generated for that cluster. Since various ways for generating GMM models for the cluster are known in the related art, no detailed description will be provided herein. 
     Further, if two clusters are close enough in the decision tree, the two clusters can be combined for subsequent clustering. As is shown in  FIG. 6 , the “No” branches of conditions  1  and  2  are close to each other (or, they are similar), therefore they are combined and thereafter used for further clustering at condition  4 . As is readily recognizable, the distance definition system may comprise a combining unit for implementing the above branch combining operations in the decision tree. 
     For more information about GMM models, please refer to N. Kambhatla, “Local Models and Gaussian Mixture Models for Statistical Data Processing” PhD thesis, Oregon Graduate Institute of Science and Technology, January, 1996. 
       FIG. 7  depicts the training system according to embodiments of the present invention. As is shown in  FIG. 7 , the training system  700  comprises a text analysis unit  701 , a prosody prediction unit  702 , and a GMM model storing unit  703  connected to said prosody prediction unit  702 , used for storing the GMM models generated for each cluster. 
     According to embodiments of the invention, said training system  700  may also contain means for storing a series of optimization questions (not shown), means for decision making with respect to said optimization questions (not shown) and means for combining the appropriate clusters for implementing the above-mentioned decision tree. 
     The method and system on the synthesis section according to embodiments of the invention will now be described with reference to  FIG. 8 , a flowchart of a synthesizing method. The synthesizing method starts from step S 810  and then proceeds to step S 820 . In step S 820 , the cluster of the unit that is to be synthesized (for example, it can be a character contained in the text) is determined so as to determine the GMM model thereof. The cluster can be determined, for example, through a series of questions in the decision tree, so as to find the GMM model corresponding therewith from the GMM model base. Next, in step S 830 , the distance between the candidate samples in the cluster and the found GMM model is calculated. One possible method of calculation is detailed below. After calculating the distance, the sample with the smallest distance is identified as the optimal sample in step S 840  for synthesizing. Then, the method ends in step S 850 . 
     Step S 830  will be elaborated in detail now. As mentioned above, embodiments of the method of the invention involves the calculation of the distance between each unit that is to be synthesized and the GMM model thereof, and the sample with the smallest distance is the best. Said distance is also known as the target cost. After calculation is completed for each unit to be synthesized, the final synthesized speech is obtained by adding all the resulting units that have the smallest distance. According to embodiments of the present invention, said cost can be calculated by employing dynamic programming. That is, to find the global optimized path through local optimized cost function estimation. 
     According to embodiments of the invention, a transition cost can be calculated in addition to said target cost. Target cost means the distance between a unit that is to be synthesized and the GMM model thereof. The speech parameters of two consecutive synthesizing units need to satisfy certain transition relationship. Only matched unit can achieve a high degree of naturalness, and a transition model depicts this transition relationship from a modeling perspective. 
     An evaluation of the transition features of the speech parameters of two consecutive synthesizing units in the current transition model, that is, the distance between the transition feature and the current transition model, is known as the transition cost. This distance can also be interpreted as a GMM model distance. 
     As shown in  FIG. 9  with the solid lines, the cost of each possible path can be attained by the accumulation of the target cost of each node and the transition cost between two neighboring nodes in the path. After all of the possible paths are evaluated, the global optimized path is generated with the smallest cost. 
     As shown in  FIG. 9 , assuming that C( 1 , x) represents the character in the previous text, C( 2 ,x) “ ” and C( 3 , x)“—” and so on. According to an embodiment of the invention, the voice output can be obtained by choosing only the smallest target cost of each unit to be synthesized and directly adding the units with the smallest target costs together. However, according to another embodiment of the invention, the transition cost may be taken into account as well. In  FIG. 9 , the path C( 1 ,  2 )-C( 2 , m 2 )-C( 3 ,  1 ) is considered the path with the smallest target cost plus transition cost. 
     The synthesizing process of the invention may be implemented through the synthesizing system  1000  shown in  FIG. 10 . The synthesizing system  1000  comprises a cluster determining unit  1001  used for determining the cluster of the unit that is to be synthesized so as to determine the corresponding GMM model from the GMM model base. After the determination of the GMM model, a distance calculating unit  1002  is used to calculate the distance between the candidate samples in the cluster and the found GMM model. Then, an optimizing unit  1003  is to evaluate the resulting distances so as to find the unit with the smallest distance. Said unit with a smallest distance is output to a synthesizing unit  1004  to form the physical voice. 
     In addition, said distance calculating unit  1002  may also comprise a target cost calculating unit and a transition cost calculating unit which are not shown. 
     The distance definition based on GMM is illustrated above. There are two typical scenarios to use the definition. One is to evaluate the distance between a given sample and a given cluster, which is the task of unit-selection based approach, and the other is to predict the explicit phonetic parameters through searching in the space of the given probability distributions. 
     The steps to apply the definition for unit selection in a TTS system are listed as follow: 
     (In the Training Process) 
     1. Extracting phonetic parameters and its context information from the labeled corpus; 
     2. Context equivalent clustering of phonetic parameters and the distance among phonetic parameters are given by GMM based distance definition; 
     3. Generating GMM to describe the probability distribution of each cluster generated in step 2. 
     (In the Synthesis Process) 
     4. Getting context information of each phonetic segment (that is, the unit to be synthesized) from the result of the text analysis unit; 
     5. Finding the context equivalent cluster of each segment, which is corresponding to a GMM; 
     6. Evaluating all of the candidates of the segment by GMM based distance definition; 
     7. Finding overall optimized candidate sequence based on distances given in step 6 and criteria of overall optimization such as dynamic programming; 
     8. Speech synthesis to generate physical voice. 
     The steps to apply the definition for explicit prediction are listed as follow: 
     (In the Training Process) 
     1. Extracting phonetic parameters and its context information from the labeled corpus; 
     2. Context equivalent clustering of phonetic parameters and the distance among phonetic parameters are given by GMM based distance definition; 
     3. Generating GMM to describe the probability distribution of each cluster generated in step 2; 
     (In the Synthesis Process) 
     4. Getting context information of each phonetic segment (that is, the unit to be synthesized) from the result of text analysis component; 
     5. Finding the context equivalent cluster of each segment, which is corresponding to a GMM; 
     6. In the space of the mixture model sequence, searching the best values based on the distance definition and criteria of overall optimization, and the sequence of best values is regarded as the explicit prediction; 
     7. Synthesis according to the explicit prediction given in step 6. 
     In order to implement the above operations, said cluster determining unit  1001  can further comprise a prosody annotation information acquiring means for acquiring the descriptive prosody annotation information of the unit to be synthesized; finding means for finding the cluster of each unit to be synthesized, said cluster corresponding to a GMM model; and means for searching for the optimal value based on the distance definition and the overall optimal criteria in the space of the GMM mixture model series so that the optimal series is used as the explicit prediction of the GMM model. 
     Correspondingly, the distance calculating unit  1002  can further comprise a prosody annotation information acquiring means for acquiring the descriptive prosody annotation information of the unit to be synthesized; finding means for finding the cluster of each unit to be synthesized, said cluster corresponding to a GMM model; and candidate evaluating means for evaluating all the candidates of the unit to be synthesized through the GMM-based distance definition. Meanwhile, the optimizing unit  1003  can further comprise a means for acquiring the overall optimal candidate series based on the distance given in the evaluation steps and the overall optimal criteria for subsequent voice synthesizing. 
       FIGS. 11 and 12  present illustrative configurations of the cluster determining unit  1001 , the distance calculating unit  1002 , and the optimizing unit  1003 . It should be noted that, the various means can have different ways for implementation, for example, by using the computer program code unit or electronic logic circuit, which is within the comprehension of those skilled in the art, and therefore detailed explanation will be omitted. 
     The essential of GMM based distance definition is to precisely simulate the probability distribution of a defined cluster in data for TTS, and then give the distance between an isolated sample and the cluster, which is very critical for unit selection based approach. Another advantage of GMM based distance definition is that some mature algorithms of tolerance, adaptation and so on can be smoothly deployed in statistical technologies of TTS. 
     In the TTS training and synthesizing according to embodiments of the invention, a decision tree, GMM, and dynamic programming may be combined to form a unit selection based TTS system, wherein GMM is used to describe the prediction of the target for each node in the synthesis sequence and the prediction of transition between the neighboring nodes. 
     The main points in the combination lie in:
         At first, a decision tree based clustering algorithm is used to split all of the prosody vectors of segments in corpus into reasonable classes. The number of classes depends on the pre-defined criteria and the amount of data in corpus.   For each class, a GMM is trained based on the data in it.   The cost functions in dynamic programming are changed to be log probability function, which means that the global optimized path is the one with largest accumulation log probability values. It may be regarded as the negative operation of cost functions.   GMMs of prosody targets for each node generate target log probability functions. Target prediction is a popular approach in some TTS systems, and GMMs of prosody transitions for two neighboring nodes may generate transition log probability functions.       

     The concept of prosody transitions is introduced below. As mentioned before, target prosody is broadly used, which is a natural way to predict the expectation of each segment and do selection based on the prediction. The biggest challenge may be the data dispersing problem. For example,  FIG. 13  is a graph of all the data in a leaf of a pitch tree. The range appears large and the distribution appears average. Although it is easy to give out target probability prediction through GMM model for targets, it is difficult to expect that only target models can get good selection result. 
     Smoothing criteria may be used to resolve some problems, but not all, and the most important issue is that some cases become bad with simple smoothing criteria.  FIG. 14  elaborates the phenomena more in detail. The two parameters between neighboring units may exist at a reasonable jump, and the amplitude values of jumps are context dependent. 
     Probability model for transition prosody is proposed to model the variety between the two neighboring segments. There are many transition related prosody parameters, for example, difference of log pitch, log duration and loudness values between the two segments. It is natural that the transition models generate the transition probability output in the dynamic programming searching scheme. 
     According to embodiments, the probability model of transition prosody integrated into the combination of decision tree, GMM, and dynamic programming. On the one hand, all of the segments in corpus can be used to train a target probability prediction tree and a single transition probability trees, which means that there are no data sparse problems in probability model building. Because of transition model, even though there are still data dispersing problems, the influence is partly removed, which makes the predicted prosody more stable and more reasonable. 
     The foregoing description of the exemplary embodiment of the invention has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. May modifications and various are possible in light of the above teachings. For example, this invention can be implemented by means of software, hardware or the combination thereof. It is intended that the scope of the invention be limited not with this detailed description, but rather determined by the appended claims.