Abstract:
A text to-speech method and a multi-lingual speech synthesizer using the method are disclosed. The multi-lingual speech synthesizer and the method executed by processor are applied for processing a multi-lingual text message in a mixture of a first language and a second language into a multi-lingual voice message. The multi-lingual speech synthesizer comprises a storage device configured to store a first language model database, second language model database a broadcasting device configured to broadcast the multi-lingual voice message, and a processor, connected to the storage de ice and the broadcasting device, configured to ex cute the method disclosed herein.

Description:
RELATED APPLICATIONS 
       [0001]    This application claims priority to Taiwan Application Serial Number 104123585, filed Jul. 21, 2815, and Serial Number 104137212, filed Nov. 1, 2015, the entirety of which is herein incorporated by reference. 
       BACKGROUND OF THE INVENTION 
       [0002]    Field of the Invention 
         [0003]    The present disclosure relates to a text to speech method and, more particularly, to a text to speech method and a synthesizer for processing a multi-lingual text message into a multi-lingual voice message. 
         [0004]    Description of the Related Art 
         [0005]    With the globalization, multiple languages are usually blended in conversation. For example, professional field terms, terminology, foreigner names, foreign geographical names, and foreign specific terms that are not easily translated, would be blended in the local language. 
         [0006]    In general TTS (Text-To-Speech) methods are usually used for a single language, and a voice message is searched in a corresponding language database, and then converted to a voice message corresponding to the language. However, the conventional TTS cannot effectively process the text message with two or more languages, since the databases do not include the corresponding voice message with two of more language. 
       SUMMARY OF THE INVENTION 
       [0007]    According to a first aspect of the present disclosure, a text-to-speech method executed by a processor for processing a multi-lingual text message in a mixture of a first language and a second language into a multi-lingual voice message, cooperated with a first language model database ha lug a plurality of first language phoneme labels and first language cognate connection tone information and a second language model database having a plurality of second language phoneme labels and second language cognate connection tone information, comprises: separating the multi-lingual text message into at least one first language section and at least one second language section; converting the at least one first language section into at least one first language phoneme label and converting the at least one second language section into at least one second language phoneme label; looking up the first language model database using the at least one first language phoneme label thereby obtaining at least one first language phoneme label sequence, and looking up the second language database model using the at least one second language phoneme label thereby obtaining at least one second language phoneme label sequence; assembling the at least one first language phoneme label sequence and at least one second language phoneme label sequence into a multi-lingual phoneme label sequence according to an order of words in the multi-lingual text message; producing inter-lingual connection tone information at a boundary between every two immediately adjacent phoneme label sequences, wherein every mo immediately adjacent phoneme label sequences includes one of the at least one first language phoneme label sequence and one of the at least one second language phoneme label sequence; combining the multi-lingual phoneme label sequence, the first language cognate connection tone information at a boundary between every two immediately adjacent phoneme label of the at least one first language phoneme label sequence, the second language cognate connection tone information at a boundary between every two immediately adjacent phoneme labels of the at least one second language phoneme label sequence, and inter-lingual connection tone information to obtain the multi-lingual voice message, and outputting the multi-lingual voice message. 
         [0008]    Furthermore, according to a second aspect of the present disclosure, a multi-lingual speech synthesizer for processing a multi-lingual text message in a mixture of a first language and a second language into a multi-lingual voice message, comprises: a storage device configured to store a first language model database having a plurality of first language phoneme labels and first language cognate connection tone information, and a second language model database having a plurality of second language phoneme labels and second language cognate connection tone information; a broadcasting device configured to broadcast the multi-lingual voice message a processor, connected to the storage device and the broadcasting device, configured to: separate the multi-lingual text message into at least one first language section and at least one second language section; convert the at least one first language section into at least one first language phoneme label and converting the at least one second language section into at least one second language phoneme label; look up the first language model database using the at least one first language phoneme label thereby obtaining at least one first language phoneme label sequence, and look up the second language database model using the at least one second language phoneme label thereby obtaining at least one second language phoneme label sequence; assemble the at least one first language phoneme label sequence and at least one second language phoneme label sequence into a multi-lingual phoneme label sequence according to an order of words in the multi-lingual text message; produce inter-lingual connection tone information at a boundary between every two immediately adjacent phoneme label sequences, wherein every two immediately adjacent phoneme label sequences includes one of the at least one first language phoneme label sequence and one of the at least one second language phoneme label sequence; combine the multi-lingual phoneme label sequence, the first language cognate connection tone information at a boundary between every two immediately adjacent phoneme label of the at least one first language phoneme label sequence, the second language cognate connection tone information at a boundary between every two immediately adjacent phoneme labels of the at least one second language phoneme label sequence, and inter-lingual connection tone information to obtain the multi-lingual voice message, and output the multi-lingual voice message to the broadcasting device. 
         [0009]    Further scope of applicability of the present disclosure will become apparent from the detailed description given hereinafter. However, it should be understood that the detailed description and specific examples, while indicating preferred embodiments of the disclosure, are given by way of illustration only, since various changes and modifications within the spirit and scope of the disclosure will become apparent to those skilled in the art from this detailed description. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0010]    These and other features, aspects, and advantages of the present disclosure will become better understood with regard to the following description, appended claims, and accompanying drawings. 
           [0011]      FIG. 1  is a block diagram showing a multi-lingual speech synthesizer in an embodiment; 
           [0012]      FIG. 2  is a flowchart of a text-to-speech method in accordance with an embodiment; 
           [0013]      FIGS. 3 and 4  illustrate a flowchart of step S 240  in accordance with an embodiment; 
           [0014]      FIG. 5  is a flowchart of step S 250  in accordance with an embodiment; 
           [0015]      FIGS. 6A-6B  illustrate the calculation of available candidates of the audio frequency data in accordance with an embodiment; 
           [0016]      FIG. 7  is a schematic diagram showing the determination of connecting paths of the pronunciation units in accordance with an embodiment; 
           [0017]      FIG. 8  is a flowchart showing a training method of a training program of the TTS method  200  in accordance with an embodiment; and 
           [0018]      FIGS. 9A-9C  show a training voice ML, voice samples SAM and the pitch, the tempo and the timbre of a mixed language after analyzing different languages in accordance with an embodiment. 
       
    
    
     DETAILED DESCRIPTION OF THE EMBODIMENTS 
       [0019]    The order of steps in embodiments is not used for restricting the sequence in execution, The device equivalent to the recombination of components in the disclosure is also within the scope of the disclosure. 
         [0020]    The wording “first” and “second” and so on do not represent the order or the sequence, which are just used for distinguish terms with the same name. The terms “include”, “comprise”, “have” are open-ended. 
         [0021]      FIG. 1  is a block diagram showing a multi-lingual speech synthesizer in accordance with an embodiment. As shown in  FIG. 1 , a multi-lingual speech synthesizer  100  includes a storage module  120 , a broadcasting device  140 , and a processor  160 . 
         [0022]    The multi-lingual speech synthesizer  100  is used for processing/converting a text message to a corresponding multi-lingual voice message, and the broadcasting device  140  outputs the multi-lingual voice message. In an embodiment, the multi-lingual speech synthesizer  100  processes a multi-lingual text message. 
         [0023]    In an embodiment, the storage module  120  stores a plurality of language model databases, e.g., LMD 1 , LMD 2 , etc., and each of the language model databases corresponds to a single language (e.g., Mandarin, English, Japanese, German, French, Spanish). Furthermore, each of the language model databases includes a plurality of phoneme labels of a single language and cognate connection tone information. In an embodiment, the multi-lingual text message blends two languages-Mandarin and English, and the storage module  120  stores a Mandarin model database LMD 1  and an English model database LMD 2 . However, the varieties of languages are not limited herein. A mixed multi-language model database for both Mandarin and English is not needed in an embodiment. 
         [0024]    The phoneme label is a minimum sound unit with a distinguishable pronunciation. In an embodiment, a word or a character includes at least one syllable, and one syllable includes at least one phoneme. In an embodiment, a Mandarin character includes one syllable, and the syllable usually includes one to three phonemes (each of phonemes is similar to a pinyin symbol). In an embodiment, an English word includes at least one syllable, each syllable includes one to several phonemes (each phoneme is similar to an English phonetic symbol). In an embodiment, each language model database includes pronunciation of phonemes and connection tone information between the phonemes for a better voice effect. The connection tone information provides a tone for connecting a preceding phoneme and a succeeding phoneme when two immediately adjacent phonemes (belongs to two immediately adjacent words or characters) are pronounced. 
         [0025]    The phoneme label is a representing-symbol facilitating the system. processing. Each of the language model databases LMD 1  and LMD 2  further stores audio frequency data including a pitch, a tempo, timbre of each phoneme label for pronunciation. In an embodiment, the pitch includes, but not limited to, the frequency of pronunciation, the tempo includes, but not limited to, speed, interval, rhythm of the pronunciation, and the timbre includes, but not limited to, pronunciation quality, mouthing shapes, and pronunciations. 
         [0026]      FIG. 2  is a flow chart showing the steps of a text-to-speech method in accordance with an embodiment. A multi-lingual text-to-speech method  200  is used for processing/converting a text message including different languages into a multi-lingual voice message. In an embodiment, the multi-lingual text-to-speech method is executed by a processor  160 , such as, but not limited to, a central processing unit (CPU), a System on Chip (SoC), an application processor, an audio processor, a digital signal processor, or a controller with a specific function. 
         [0027]    In an embodiment, the multi-lingual text message can be, but not limited to, a paragraph in an article, an input command, selected words or characters in a webpage. In an embodiment, a first language model database has a plurality of first language phoneme labels and first language cognate connection tone information, and a second language model database has a plurality of second language phoneme labels and second language cognate connection tone information. 
         [0028]    As shown in  FIG. 2 , the multi-lingual text-to-speech method  200  includes the following steps. Step S 210  is to separate the multi-lingual text message into at least one first language section and at least one second language section. in an embodiment, the processor  160  separates the multi-lingual text message into language sections according to different languages. In an embodiment, the text message “         Jason Mraz         ” is separated into three language sections such as “         ” (a Mandarin language section), “         ” (a Mandarin language section), and “Jason Mraz” (an English language section). 
         [0029]    Step S 220  is to convert the at least one first language section into at least one first language phoneme label and to convert the at least one second language section into at least one second language phoneme label. In an embodiment, each phoneme label includes audio frequency data such as, but not limited to, a pitch, a tempo, timbre of phonemes. 
         [0030]    Step S 230  is to look up the first language model database LMD 1  using the at least one first language phoneme label thereby obtaining at least one first language phoneme label sequence, and to look up the second language database LMD 2  model using the at least one second language phoneme label thereby obtaining at least one second language phoneme label sequence. 
         [0031]    In an embodiment, letter “M” represents phonemes of Mandarin, and the number represents the serial number of phonemes in Mandarin. In an embodiment, the Chinese character “         ” corresponds to two phoneme labels [M 04 ] and [M 29 ], the Chinese character “         ” corresponds to another two phoneme labels [M 09 ] and [M 25 ]. As a result, the phoneme label sequence that convened from the Mandarin language section “         ” is [M 04  M 29  M 09  M 25 ]. Similarly, the phoneme label sequence corresponding to the language section “         ” is [M 88  M 29  M 41  M 44 ]. Moreover, the phoneme label sequence corresponding to the English language section “Jason Mraz” is [E 19  E 13  E 37  E 01  E 40 ] according to the English model database LMD 2 . 
         [0032]    Step S 240  is to assemble the at least one first language phoneme label sequence and at least one second language phoneme label sequence into a multi-lingual phoneme label sequence according to an order of words (or characters) in the multi-lingual text message. 
         [0033]    In other words, the processor  160  arranges the multiple phoneme label sequences of the different language sections according to the sequence of the original multilingual text message, and assembles the arranged phoneme label sequences into a multi-lingual phoneme label sequence. In the embodiment, the three converted phoneme label sequences of the text message “         Jason Mraz         ”, i.e., [M 04  M 29  M 09  M 25 ], [E 19  E 13  E 37  E 01  E 40 ], and [M 04  M 29  M 09  M 25 ], are assembled into a multi-lingual phoneme label sequence as [M 04  M 29  M 09  M 25  E 19  E 13  E 37  E 01  E 40  M 08  M 29  M 41  M 44 ] according to the sequence of the original multi-lingual text message. 
         [0034]    In step S 250 , the processor  160  produces inter-lingual connection tone information at a boundary between every two immediately adjacent phoneme label sequences, wherein every two immediately adjacent phoneme label sequences includes one of the at least one first language phoneme label sequence and one of the at least one second language phoneme label sequence. In an embodiment, the processor  160  looks up the language model databases LMD 1  and LMD 2  to obtain inter-lingual connection tone information for each two immediately adjacent phoneme labels. An embodiment of the detailed process is described hereinafter. 
         [0035]    In step S 260 , the processor  160  combines the multi-lingual phoneme label sequence, the first language cognate connection torte information at a boundary between every two immediately adjacent phoneme label of the at least one first language phoneme label sequence, the second language cognate connection tone information at a boundary between every two immediately adjacent phoneme labels of the at least one second language phoneme label sequence, and inter-lingual connection tone information to obtain the multi-lingual voice message and in step S 270 , the multi-lingual voice message is outputted. 
         [0036]    For better voice effect, in an embodiment, the step S 240  of the text-to-speech method in  FIG. 2  further includes steps S 241 -S 245 . which is as showed in  FIG. 3 . 
         [0037]    As shown in  FIG. 3 , in S 241 , the processor  160  divides the assembled multi-lingual phoneme label sequence jaw a plurality of first pronunciation units, and each of the plurality of first pronunciation units is in a single language and includes consecutive phoneme labels of a corresponding one of the at least one first language phoneme label sequence and the at least one second language phoneme label sequence. 
         [0038]    Then, step S 242  is executed on each of the first pronunciation units. In step S 242 , the processor  160  determines whether a number of available candidates for a corresponding one of the first pronunciation units in a corresponding, one of the first language model database and the second language model database is equal to or more than a predetermined number corresponding to the one of the first pronunciation units. When the number of available candidates for each of the first pronunciation units in the corresponding one of the first language model database and the second language model database is determined equal to or more than the corresponding predetermined number, then the processor  160  executes the step S 243  to calculate a join cost of each candidate path, wherein each candidate path passes through one of the available candidates of each of the first pronunciation units. In step S 244 , the processor  160  determines a connecting path between every two immediately adjacent first pronunciation units based on the join cost of each candidate path. 
         [0039]    Further, in an embodiment, in the step S. 244  the processor  160  further determines a connecting path between a selected one of the available candidates in a front one of two immediately adjacent first pronunciation units and a selected one of the available candidates in a rear One of two immediately adjacent first pronunciation units, wherein the selected one of the available candidates in the front one of two immediately adjacent first pronunciation units and the selected one of the available candidates in the rear one of two immediately adjacent first pronunciation units are both located in one of the candidate paths that has a lowest join cost. 
         [0040]    However, after step  242 , when the number of available candidates for any one or one of the first pronunciation units in the corresponding one of the first language model database and the second language model database is determined to be less than the corresponding predetermined number, a subset (indicated as A in  FIG. 3 ) of steps S 246  and S 247  in an embodiment as showed in  FIG. 4  is proceeded. 
         [0041]    In step S 246  in  FIG. 4 , the processor  160  further divides the one or ones of the first pronunciation units into a plurality of second pronunciation units, a length of any one of the second pronunciation units is shorter than a length of a corresponding one of the first pronunciation units. In step S 247 , for each of the second pronunciation units, the processor  160  further determines whether a number of available candidates for a corresponding one of the second pronunciation units in a corresponding one of the first language model database and the second language model database is equal to or more than a predetermined number corresponding to the one of the second pronunciation units. 
         [0042]    In other words, the subset of steps S 246  and S 247  is repeated if the number of available candidates for any one or ones of the first pronunciation units (or the second pronunciation units and so on.) in the corresponding one of the first language model database and the second language model database is determined to be less than the corresponding predetermined number in step  242 , until the number available candidates is determined equal to or more than the corresponding predetermined number, and then a join cost of each candidate path is calculated in step  243 . 
         [0043]    In an embodiment, a multi-lingual text message “                   Boston University          ” is divided into several first pronunciation units such as audio frequency date “         ”, “         ”, “         ”, “         ”, “Boston University”, “         ”. The processor  160  then determines whether a number of available candidates for these first pronunciation units in a corresponding one of the first language model database and the second language model database is equal to or more than a predetermined number corresponding to the one of the first pronunciation units. 
         [0044]    In an embodiment, assuming that the predetermined number of available candidates for the first pronunciation unit “         ” is ten, if only five available candidates for the first pronunciation unit “         ” in the first language model database LMD 1  are available, this means that the number of available candidates in the first language model database LMD 1  is less than the corresponding predetermined number, and then second pronunciation units with a shorter length than the first pronunciation unit “         ”are divided from the first pronunciation unit “         ”, as step  246  in  FIG. 4 . 
         [0045]    In an embodiment, the predetermined number for each of the second pronunciation units is the same as the predetermined number for the corresponding first pronunciation unit. In another embodiment, the predetermined number for each of the second pronunciation units can be set differently from the predetermined number for the corresponding first pronunciation unit. In this embodiment, the first pronunciation unit “         ” is divided into two second pronunciation “         ” and “         ”, and 280 available candidates for “         ” and 56 available candidates for “         ” are found in the first language model database LMD 1 , respectively. For example, in this embodiment, the predetermined number of available candidates for each of the second pronunciation units “         ” and “         ” is ten. That means, the number of available candidates corresponding to each of the second pronunciation units “         ” and “         ” is more than the corresponding predetermined number, and then step S 243  is consequently executed. For a better speech effect, the first pronunciation unit is further divided into shorter second pronunciation units until enough available candidates are found in the corresponding language database. 
         [0046]    As shown in  FIG. 5 , the step S 250  of producing inter-lingual connection tone information at a boundary between every two immediately adjacent phoneme label sequences further includes a subset of steps in an embodiment. The connection relationships between the phoneme labels of the pronunciation unit of a same language are stored in the language model databases LMD 1  and LMD 2 . Taking the multi-lingual phoneme label sequence [M 04  M 29  M 09  M 25  E 19  E 13  E 37  E 01  E 40  M 08  M 29  M 41  M 44 ] for the text message “         Jason Mraz         ” as an example again, the cognate connection tone information for connecting [M 04  M 29 ] is stored in the Mandarin model database LMD 1 , which is represented as L[M 04 , M 29 ], the cognate connection tone information for [M 29  M 09 ] is represented as L[M 29 , M 09 ], and so on. The cognate connection tone information for any two adjacent phoneme labels of Mandarin is stored in the language model database LMD 1 . In an embodiment, the cognate connection tone information for the adjacent phoneme labels [E 19  E 13 ] is also pre-stored in the English model database LMD 2 , and so on. 
         [0047]    Since each of the language model databases LMD 1  and LMD 2  stores information of the same language information, respectively, the inter-lingual connection tone information across two languages for the multi-lingual phoneme label sequence [M 04  M 29  M 09  M 25  E 19  E 13  E 37  E 01  E 40  M 08  M 29  M 41  M 44 ] (such as the inter-lingual connection tone information for [M 25  E 19 ] and the connection tone information for [E 40  M 08 ]) will not be found in conventional TTS method. 
         [0048]    The connection tone information between each phoneme label provides the fluency, the consistency, and the consecutiveness of the pronunciation. Therefore, in an embodiment, the processor  160  generates inter-lingual connection tone information at a boundary of any two phoneme labels between two different languages according the step S 250 , which is illustrated in detail hereinafter. 
         [0049]      FIG. 5  is a flow chart showing a method for producing the inter-lingual connection tone information at a boundary between the first language and the second language in an embodiment. As shown in  FIG. 5 , the step S 250  further includes steps S 251 -S 252 . 
         [0050]    In step S 251  of  FIG. 5 , the processor replaces a first phoneme label of the at least one second language phoneme label sequence with a corresponding phoneme label of the first language phoneme labels which has a closest pronunciation to the first phoneme label of the at least one second language phoneme label sequence 
         [0051]    In an embodiment, in the multi-lingual text message “         Jason Mraz         ”, the first boundary between the first language and the second language is the boundary between “         ” and “Jason”. In this embodiment. Mandarin is the first language, 
         [0052]    English is the second language, and the Mandarin text “         ” (corresponding to the phoneme labels [M 09  M 25 ]) appears in front of the English text “Jason” (corresponding to the phoneme labels [E 19  E 13 ]). That is, the first boundary at the last phoneme label of the language section of the first language and the first phoneme label of the language section of the second language, in the embodiment, is between the phoneme labels [M 25 ] and [E 19 ]. 
         [0053]    According to step S 251 , the first phoneme label [E 19 ] in the language section of the second language (English in the embodiment) is replaced by a phoneme label in the first language (Mandarin in the embodiment) with the closest pronunciation. In an embodiment, the. phoneme “Ja” (corresponding to the phoneme label [E 19 ]) in English is replaced with the phoneme “         ” (Pronounced as “Ji”) (corresponding to the phoneme label [M 12 ]) in Mandarin, in the embodiment, the phoneme label [E 19 ] of the phoneme “Ja” in English is replaced with a phoneme label [M 12 ] of the phoneme “         ” in Mandarin. 
         [0054]    Furthermore, in the same sample text, the second cross language boundary is the boundary between “Mraz” (corresponding to the phoneme labels [E 37  E 01  E 40 ]) and “         ” (corresponding to the phoneme labels [MOS M 29 ]). That is, the second boundary between the last phoneme label of the language Section of the second language and the first phoneme label of the language section of the first language, in this embodiment, is between the phoneme labels [E 40 ] and [MOS]. Then, the phoneme label [M 08 ] of the phoneme “         ” in Mandarin is replaced with a phoneme label [E 21 ] of the phoneme “le” in English, which is the closest pronunciation to the phoneme label [M 08 ] of the phoneme “         ”. 
         [0055]    Then, in step S 252 , the processor  160  looks up the first language model database LDM 1  using the corresponding phoneme label of the first language phoneme labels thereby obtaining a corresponding cognate connection tone information of the first language model database LDM 1  between a last phoneme label of the at least one first language phoneme label sequence and the corresponding phoneme label of the first language phoneme labels, wherein the corresponding cognate connection tone information of the first language model database LMD 1  serves as the inter-lingual connection tone information at the boundary between the one of the at least one first language phoneme label sequence and the one of the at least one second language phoneme label sequence. 
         [0056]    Specifically, in the above embodiment with the first boundary, the cognate connection tone information L[M 25  M 12 ] is found in the first language model database LMD 1  of the first language according to the last phoneme label of the first language at the first boundary and the replacing phoneme label [M 25  M 12 ], Then, the cognate connection tone information L[M 25  M 12 ] is regarded as the inter-lingual connection tone information at the first boundary. For the second boundary, the cognate connection tone information [E 40  E 21 ] can be found in the second language model database LMD 2  according to the last phoneme label of the second language at the second boundary and the closest replacing phoneme label [E 40  E 21 ]. Then, the cognate connection tone information L[E 40  E 21 ] is regarded as the inter-lingual connection tone information at the second boundary. 
         [0057]    The way of calculating available candidates of the audio frequency data is illustrated accompanying  FIGS. 6A and 6B  in an embodiment. 
         [0058]    As shown in  FIG. 6A , in the embodiment, the first pronunciation unit is “                   ”, and the pitch, the tempo and the timbre of each character corresponding to the first pronunciation unit “         ”are searched in the first language model database LMD 1 . The pitch includes, but not limited to, the frequency of phonation, the tempo includes, but not limited to, duration, speed, interval, and rhythm of the pronunciation, and the timbre includes, but not limited to, pronunciation quality, mouthing shapes, and pronunciation positions.  FIGS. 6A and 6B  are schematic diagrams showing that the pitch is compared to a benchmark average value according to an embodiment. 
         [0059]    In the embodiment, curves of the pitch and duration of the tempo of a pronunciation unit are represented by a one-dimensional Gaussian model, respectively. In the embodiment, the one-dimensional Gaussian model for the pitch is a statistical distribution of the pronunciation unit under different frequencies. The one-dimensional Gaussian model for the duration is a statistical distribution of the pronunciation unit under different time durations (such millisecond, ms) 
         [0060]    In the embodiment, the mouthing shape representing the timbre is established by multiple Gaussian mixture models. In an embodiment, the Gaussian mixture models are established by a Speaker Adaptation method to record the mouthing shapes representing the timbre, and then relative reliable mouthing shapes are established corresponding to the input text message. The Speaker Adaptation technology includes following steps: establishing a general module for all phonemes of one language according to pronunciation data of different speakers of this language; after the general module for all phonemes of this language is established, extracting a mouthing shape parameter of the required pronunciation from a recorded mixed-language file; moving the general modules of the phonemes to the sample of extracting the mouthing shape parameter, and the modules after moved are adapted models. Detailed steps and the principle of Speaker Adaptation technology are disclosed in “Speaker Verification Using Adapted Gaussian Mixture Models” on the year of 2000 in the journal “Digital Signal Processing” by Reynolds, Douglas A. However, the way of establishing the mouthing shape is not limited to the Speaker Adaptation technology. 
         [0061]    In the embodiment, a benchmark average frequency Pavg 1  of all pitches for the first pronunciation unit “         ” in the first language model database LMD 1  is obtained. In the embodiment, the average frequencies of the six Chinese characters are 100 Hz, 146 Hz, 305 Hz, 230 Hz, 150 Hz, and 143 Hz, respectively. This group of the benchmark average frequency Pavg 1  is used as the target audio frequency data, which is the reference in the subsequent selection. 
         [0062]    Then, 168 groups of the pitch frequency data PAU of the first pronunciation unit “         ” are found in the first language model database LMD 1 , as showed in  FIG. 6A , PAU 1 -PAU 168 . In an embodiment, the frequency difference between the selected group of pitch frequency data and the target audio frequency (that is, the benchmark average frequency Pavg 1 ) is set to be within a predetermined range, 20% of the benchmark average frequency Pavg1. In the embodiment, the predetermined ranges of the target audio frequency data of the six Chinese characters are 100 HZ±20%, 100 Hz±20%, 146 Hz±20%, 305 Hz±20%, 230 Hz±20%, 150 Hz±20%, and 143 Hz±20%. The group with all the six Chinese characters having audio frequency data within the predetermined range will be the candidates (PCAND). For example, in the first group of pitch frequency data PAU 1 , the frequencies of the six Chinese characters are 175 Hz, 179 Hz, 275 Hz, 300 Hz, 120 Hz, and 150 Hz in sequence, which will be outside of the predetermined range, 20% of the benchmark average Frequency Pavg 1 . In fact, among the 168 PAU groups, only two available candidate frequency data, PAU 63  and PAU 103  are within the determined range. However, assuming that the predetermined number of the first pronunciation unit is  10 , the number of available candidates (i.e., 2) (PCAND: PAU 63  and PAU 103 ) is not equal to or more than the predetermined number (i.e., 10). Therefore, the first pronunciation unit needs to be divided into a plurality of second pronunciation units that are shorter than the first pronunciation unit for more candidates. 
         [0063]    The first pronunciation unit “         ” is then divided into two second pronunciation units, “         ” and “         ”. One of the second pronunciation units, “         ”, is taken as an example for more explanation. As showed in  FIG. 6B , in an embodiment, the pitch average frequency Pavg 2  of the second pronunciation unit, “                   ”, is obtained in the first language model database LMD 1 . In an embodiment, the average frequencies of the second pronunciation unit “         ” are 305 Hz, 230 Hz, 150 Hz, and 143 Hz in sequence. The group of the benchmark average frequency Pavg 2  is the reference in the subsequent candidates&#39; determination. 
         [0064]    Then, the pitch frequency data PAU that correspond to the second pronunciation unit “         ” are searched in the first language model database LMD 1 , and 820 groups, PAU 1 -PAU 820 , are matched. In an embodiment, in the first group of pitch frequency data PAU 1 , the frequencies of the four Chinese characters are 275 Hz, 300 Hz, 120 Hz, and 150 Hz in sequence. Then, the pitch frequency data is determined from the groups of pitch frequency data PAU 1 -PAU  820  and the audio frequency data is, the benchmark average frequency Pavg  2 ) is assumed to be within a predetermined range (e.g., 20% of the benchmark average frequency Pavg  2 ). In the embodiment, the number of available candidate frequency data PCAND whose pitch frequency data is within the predetermined range is 340. The number of available candidates for the target audio frequency data is therefore enough, and the length of the second pronunciation unit is proper. Therefore, it is not necessary to divide the second pronunciation unit further into shorter pronunciation units. The range above or below the benchmark average frequency is adjustable, which is not limited to the range 20%. 
         [0065]    in the embodiment in  FIGS. 6A and 6B , available candidate audio frequency data is selected according to the pitch frequency data In another embodiment, the available candidate audio frequency data is selected according to a combination of a weigh of the pitch, the tempo, and the timbre. 
         [0066]    In an embodiment, the target audio frequency data AUavg is represented as: 
         [0000]      AUavg=αPavg+βTavg+γFavg
 
         [0000]    Wherein Pavg represents an average frequency of the pitch, Tavg represents an average duration of the tempo, Favg represents an average mouthing shape of the timbre. In an embodiment, the mouthing shape is represented by a multi-dimensional matrix. In an embodiment, the mouthing shape is represented by a Mel-frequency cepstral coefficient (MFCC). α, β, and γ represent the weight of Pavg, Tavg, and Favg, respectively. Each of the value of α, β, and γ larger than 0, and the sum of α, β, and γ is 1. In an embodiment, available candidate audio frequency data is determined according to the target sound information AUavg and the result with the weight on the pitch, the tempo, and the timbre of the audio frequency data in the language model database LMD 1 . 
         [0067]      FIG. 7  is a schematic diagram showing the determination of connecting paths of the pronunciation units in an embodiment. 
         [0068]    As shown in  FIG. 7 , in an embodiment, the text message is finally separated in to a pronunciation unit PU 1  (such as a Chinese character), a pronunciation unit PU 2  (such as a word), and a pronunciation unit PU 3  (such as a phase). In the embodiment, tbur available candidate audio frequency data AU 1   a -AU 1   d  corresponding to the pronunciation unit PU 1  are obtained in the language model databases LMD 1  and LMD 2  two available candidate audio frequency data AU 2   a -AU 2   b  corresponding to the pronunciation unit PU 2  are obtained in the language model databases LMD 1  and LMD 2 ; and three available candidate audio frequency data AU 3   a -AU 3   c  corresponding to the pronunciation unit PU 3  are obtained in the language model databases LMD 1  and LMD 2 . 
         [0069]    Connecting paths L 1  from available candidate audio frequency data AU 2   a  and AU 2   b  to available candidate audio frequency data AU 1   a -AU 1   d  are obtained in the language model databases LMD 1  and LMD 2 , and connecting paths L 2  from available candidate audio frequency data AU 2   a  and AU 2   b  to available candidate audio frequency data AU 3   a -AU 3   c  are obtained in the language model databases LMD 1  and LMD 2 . 
         [0070]    Each of available candidate paths includes a fluency cost, and each of the connecting paths includes a fluency cost. In step S 254 , a connecting path with a minimum fluency cost is selected from different combinations of the connecting paths L 1  and L 2  according to the sum of the fluency cost of the three pronunciation units PU 1 -PU 3  and the fluency cost of the connecting paths L 1  and L 2 . As a result, the pronunciation of the selected connecting path is most fluent. 
         [0071]    The formula of calculating the minimum fluency cost is as follows. 
         [0000]      Cost=α·all candidate audio frequency data of each of the pronunciation units,  C   Target ( U   i   j ) +β·all candidate audio frequency data of each two adjacent pronunciation units,  C   Spectrum ( U   i   j   , U   i+1   k ) +γ·all candidate audio frequency data of each two adjacent pronunciation units,  C   Pitch ( U   i   j   , U   i+1   k ) +δ·all candidate audio frequency data of each two adjacent pronunciation units,  C   Duration ( U   i   j   , U   i+1   k ) +ε·all candidate audio frequency data of each two adjacent pronunciation units,  C   Intensity ( U   i   j   , U   i+1   k )
 
         [0072]    Wherein represents all available candidate audio frequency data of each of the pronunciation units, U i+1   k  represents all available candidate audio frequency data of an adjacent pronunciation unit,. 
         [0073]    The sum fluency cost equals to the sum of the target cost value (such as C Target (U i   j ) in the following formula) of available candidate audio frequency data of all pronunciation units, the spectrum cost value (such as C Spectrum (U i   j , U i+1   k )) of available candidate audio frequency data between the two adjacent pronunciation units, the pitch cost value (such as C Pitch (U i   j , U i+1   k )) of available candidate audio frequency data between the two adjacent pronunciation units, the tempo cost value (such as C Duration (U i   j , U i+1   k )) of available candidate audio frequency data between the two adjacent pronunciation units, and the intensity cost value such as C Intensity  (U i   j , U i+1   k )) of available candidate audio frequency data between the two adjacent pronunciation units. In the following formula, α, β, γ, δ, and ε represent the weight of the target cost value, the spectrum cost value, the pitch cost value, the tempo cost value, and the intensity cost value, respectively. The fluency cost at different combinations along the path L 1  and L 2  is compared, and then the sum of the fluency cost with the minimum value is selected as the final sound information. 
         [0074]    The fluency cost at each path selection is calculated according to the above formula, and the fluency cost with the lowest fluency cost is obtained. In an embodiment, the fluency cost of the path from available candidate audio frequency data AU 1   c  through available candidate audio frequency data AU 2   b  to available candidate audio frequency data AU 3   a  is minimum, and then the available candidate audio frequency data AU 1   c , the available candidate audio frequency data AU 2   b,  and the available candidate audio frequency data AU 3   a  at the path is selected as the final audio frequency data in the text-to-speech method. 
         [0075]    Then, according to step S 260  in  FIG. 2 , the processor  160  generates the multi-lingual voice message by arranging and combining the audio frequency data (such as the audio frequency data AU 1   c , AU 2   b,  and AU 3   a ) of the pronunciation units. And the multi-lingual voice message is output by a broadcasting device  140  as step S 270  in  FIG. 2 , and then the sound output in the TTS method  200  is complete. In the embodiment, the broadcasting device  140  is, but not limited to, a loudspeaker, and/or a handset. 
         [0076]    In the embodiment, each of the language model databases LMD 1  and LMD 2  is pre-established via a training program. In an embodiment, the TTS method  200  further includes a training program for establishing and training the language model databases LMD 1  and LMD 2 . 
         [0077]    As shown in  FIG. 1 , the multi-lingual speech synthesizer  100  further includes a voice receiving module  180 . In the embodiment, the voice receiving module  180  is built in the multi-lingual speech synthesizer  100 , or independently exists outside the multi-lingual speech synthesizer  100 . In an embodiment, the voice receiving module  180  is, but not limited to, a microphone or a sound recorder. 
         [0078]    In an embodiment, the voice receiving module  180  samples at least a training voice to execute the training program for each of the language model databases LMD 1  and LMD 2 . The generated language model databases LMD 1  and LMD 2  after trained are provided to the multi-lingual speech synthesizer  100 . 
         [0079]      FIG. 8  is a flow chart showing a training method of a training program of the TTS method  200  according to an embodiment. Referring to  FIGS. 8 and 9A-9C , in the training program as showed in  FIG. 8 , in step S 310 , the voice receiving module  180  receives at least one training speech voice in a single language.  FIGS. 9A-9C  illustrate a schematic diagram showing a training voice ML, voice samples SAM and the pitch, the tempo and the timbre of a mixed language after analyzing different languages. In the embodiment, the pitch includes, but not limited to, frequency of pronunciation, the tempo includes, but not limited to, duration, speed, interval, and rhythm of the pronunciation, and the timbre, but not limited to, includes pronunciation quality, mouthing shapes (such as MFCC), and pronunciation positions. 
         [0080]    In an embodiment, as shown in  FIG. 9A , the multi-lingual voice sample SAM for the training voice ML is obtained from a person speaking Mandarin as a native language, and the person speaking Mandarin as the native language can speak Mandarin and English fluently. Then the pronunciation blended with Mandarin and English is obtained from the person, so that the transition between Mandarin and English is smooth. Similarly, a person speaking English as a native language and speaking Mandarin and English fluently can also be chosen for the training. 
         [0081]    In an embodiment, a training voice only includes a first voice sample of Mandarin and a second voice sample of English, and the two voice samples are recorded by a person speaking Mandarin as a native language and a person speaking English as a native language, respectively. Then, in step S 320 , the pitch, the tempo, and the timbre of the two different languages in the training voice samples are analyzed,. As shown in  FIG. 9B , the mixed language training voice ML in  FIG. 9A  is separated into the voice sample SAM 1  of the first language LAN 1  and the voice sample SAM 2  of the second language LAN 2 . Then, as shown in  FIG. 9C , the pitch, the tempo, and the timbre of the voice sample SAM 1  of the first language LAN 1  and the voice sample SAM 2  of the second language LAN 2  are analyzed to get audio frequency data such as frequency, duration, and the mouthing shapes. The pitch P 1 , the tempo T 1 , and the timbre F 1  of the voice sample SAM 1  are obtained, and the pitch P 2 , the tempo  12 , and the timbre F 2  of the voice sample SAM 2  are obtained, 
         [0082]    The pitch P 1  and the pitch P 2  are frequency distributions of the voice sample SAM 1  and the voice sample SAM 2  of all pronunciation units, respectively, the horizontal axis shows deferent frequencies (the unit is Hz), and the vertical axis shows the statistical number of the samples. The tempo TI and the tempo  12  show the duration distributions of the voice sample SAM 1  and the voice sample SAM 2  of all pronunciation units, the horizontal axis shows different durations (such as ms), the vertical axis shows the statistical number of the samples. A single sample is a single frame of one phoneme of the voice sample SAM 1  or the voice sample SAM 2 . 
         [0083]    In the embodiment, the timbre F 1  and the timbre F 2  are the mouthing shapes of all pronunciation units of the voice sample SAM 1  and the voice sample SAM 2 , respectively, which are represented by multiple Gaussian mixture models as shown in  FIG. 9C , respectively. 
         [0084]    The pitch P 1 , the tempo T 1 , and the timbre F 1  of the voice sample SAM 1  of the first language LAN 1  are stored in the language model database LMD 1 , and the pitch P 2 , and the tempo T 2 , and the timbre F 2  of the voice sample SAM 2  of the second language LAN 2  are stored in the language model database LMD 2 . 
         [0085]    Next, step S 330  is to store the training speech voice that has the pitch, the tempo and the timbre of the training speech voice each falling within a corresponding predetermined range. The pitch, the tempo, or the timbre of each of languages in the training voice is compared to a benchmark range, in an embodiment, the benchmark range is a middle range of voices already-recorded, such as a range above or below two standard deviations by the average of the pitch, the tempo, or the timbre. This step includes excluding training voice samples whose pitch, tempo, or timbre is beyond the benchmark range. Consequently, the pitch, the tempo, or the timbre with extreme values are excluded, or the voice samples with great difference (for example, the pitch of samples from a person with Mandarin as the native language and that of samples from with English as the native language are large) are excluded, and then the consistency of the pitch, the tempo, and the timbre of the two languages are improved. 
         [0086]    That is, when the pitch, the tempo, or the timbre of the newly recorded training voice is far beyond the average of the already-recorded data of statistical distribution module (for example, the pitch, the tempo, or the timbre is beyond two standard deviations of the statistical distribution module, or distributes out of the predetermined range 10%-90%), the newly recorded training voice is filtered out, and then the pitch, the tempo, or the timbre (such as pronunciation too shrill or too excited) with a large difference would not affect the consistency of available candidate audio frequency data in the language model databases. At last, the training speech voice is stored in the language model database LMD 1  or LMD 2  according to the language. 
         [0087]    As illustrated in the above embodiments, a multi-lingual text message is converted into a multi-lingual voice message such that the fluency, the consistency, and the consecutiveness of the pronunciation are improved. 
         [0088]    Although the present disclosure has been described in considerable detail with reference to certain preferred embodiments thereof, the disclosure is not for limiting the scope. Persons having ordinary skill in the art may make various modifications and changes without departing from the scope. Therefore, the scope of the appended claims should not be limited to the description of the preferred embodiments described above.