Abstract:
A soundalike system to improve speech synthesis by training a text to speech engine on a voice like the target speakers voice

Description:
CLAIM OF PRIORITY 
       [0001]    This patent application claims priority from U.S. Provisional Patent Application No. 62/314,759, filed on Mar. 29, 2016 in the U.S. Patent and Trademark Office, the disclosure of which is incorporated herein by reference in its entirety. 
     
    
     BACKGROUND 
     1. Field 
       [0002]    Embodiments herein relate to a method and apparatus for exemplary speech synthesis. 
       2. Description of Related Art 
       [0003]    Typically, speech synthesis is accomplished through the use of a speech synthesizer which generates speech through one or more pre-programmed voices. 
       SUMMARY 
       [0004]    Embodiments of the present application relate to speech synthesis using a voice that is similar to the target speaker&#39;s voice. 
         [0005]    Speech synthesis is the artificial production of human speech. A computer system used for this purpose is called a speech computer or speech synthesizer and can be implemented in software or hardware. A text-to-speech (TTS) system converts normal language text into speech; other systems render symbolic linguistic representations like phonetic transcriptions into speech. 
         [0006]    A text-to-speech system, or engine, is composed of two parts: a front-end and a back-end. The front-end has two major tasks. First, it converts raw text containing symbols like numbers and abbreviations into the equivalent of written-out words. This process is often called text normalization, pre-processing, or tokenization. The front-end then assigns phonetic transcriptions to each word, and divides and marks the text into prosodic units, like phrases, clauses, and sentences. The process of assigning phonetic transcriptions to words is called text-to-phoneme or grapheme-to-phoneme conversion. Phonetic transcriptions and prosody information together make up the symbolic linguistic representation that is output by the front-end. The back-end often referred to as the synthesizer then converts the symbolic linguistic representation into sound. In certain systems, this part includes the computation of the target prosody (pitch contour, phoneme durations) which is then imposed on the output speech. 
         [0007]    To synthesize a target speakers voice, a TTS must first train on the target voice. The speaker speaks many hours of utterances spanning all the possible language information, e.g. phonemes, diphones, triphones, etc. For optimal training, the speaker reads these utterances from text provided to him/her. The speaker reads an utterance and an Automatic Speech Recognizer (ASR) converts the audio into text. This text is matched with the actual text provided to him and label matching is done to check the correctness and the quality of the utterance. Further processing is done on the audio to get the right sampling rate and noise free audio signal. This is done for all audio and once ready, the audio is supplied to an algorithm to build a model based on distinctive characteristics (features) such as pitch, vocal tract information, formants, etc. These features are extracted and a mathematical (probabilistic) model is constructed based on well-known algorithms. 
         [0008]    When an incoming target voice is to be synthesized, the target voice is received by an ASR which outputs text. The text is broken down to units present in the model trained earlier, the closest unit is obtained along with the audio part of that unit with prosody. This is done for all the units in the input string and once the audio attaches to the unit, a stitching process is performed to combine these audio parts along with the units into an audio clip which must sound natural as if the actual human is talking 
         [0009]    The problem inherent in training a TTS is that a TTS requires the speaker to spend dozens of hours, if not more, to properly train the TTS. Specifically, the TTS needs enough speech to adequately synthesize the target speaker&#39;s voice. 
         [0010]    The solution herein is to select a voice, aka the soundalike voice, from a database of voices, wherein the soundalike voice is substantially similar to the target voice and use the soundalike voice to build the TTS voice, i.e. train the TTS. 
         [0011]    This computer system described herein is optimally configured to determine which voice from the database of voices is the most similar to the target voice. 
         [0012]    The ideal database will have voices in the language of the target speaker; a range of voices (pitch, gender, accent, etc.) is preferable. For example, a database containing a statistically significant distribution of voices is more likely to contain a good match to a speaker with a deep male voice than a database of primarily soprano female voices. This is because the identity of the target speaker is often unknown and thus a wide range of voices is more likely to find a good match. However, even when the identity of the speaker is partially known (e.g. gender), a wide distribution of voices in the database is still optimal. However, on occasion it is preferable to have a database with a narrow distribution of voices. This can occur when the target voice is constrained, e.g. male tenors. 
         [0013]    Optimally a database should contain at least 200 voices, each voice having spoken  200  sentences of 5 to 6 seconds duration per sentence. Thus a database will have 200,000 to 240,000 seconds or approximately 55 to 66 hours of voice data. 
     
    
     
       DESCRIPTION OF THE DRAWINGS 
         [0014]      FIG. 1  is a schematic diagram of the soundalike computer system. 
           [0015]      FIG. 2  illustrates a high flow diagram of the soundalike selection process. 
           [0016]      FIG. 3  illustrates a flow diagram of training the Database  125 . 
           [0017]      FIG. 4  illustrates a K-Means clustering. 
           [0018]      FIG. 5  illustrates a flow diagram of the soundalike system creating a mathematical model for the database at the cluster level and calculating the i-vector of the target voice. 
           [0019]      FIG. 6  illustrates a flow diagram of Group Selector  175  determining which group contains the soundalike voice. 
       
    
    
     DETAILED DESCRIPTION OF THE EMBODIMENTS 
       [0020]      FIG. 1  illustrates a block diagram for selecting a voice, from a database of voices, which is substantially similar to a target voice. 
         [0021]    The soundalike system in  FIG. 1  may be implemented as a computer system  110 ; a computer comprising several modules, i.e. computer components embodied as either software modules, hardware modules, or a combination of software and hardware modules, whether separate or integrated, working together to form an exemplary computer system. The computer components may be implemented as a Field Programmable Gate Array (FPGA) or Application Specific Integrated Circuit (ASIC), which performs certain tasks. A unit or module may advantageously be configured to reside on the addressable storage medium and configured to execute on one or more processors or microprocessors. Thus, a unit or module may include, by way of example, components, such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functionality provided for in the components and units may be combined into fewer components and units or modules or further separated into additional components and units or modules. 
         [0022]    Input  120  is a module configured to receive the Voice  120   b  from an Audio Source  120   a . The Audio Source  120   a  maybe be one of several sources including, but not limited to, Human  121  speaking, Streamed Speech  122  or preferentially the Database  125  containing human speech, aka voices, but may also be a live person speaking into a microphone, synthesize speech, streamed speech, etc. 
         [0023]    DB Trainer  130  is a module configured to train a database by extracting the Mel Frequency Cepstral Coefficients (MFCCs) from the Voice  120   b  in Audio Source  120   a , using the extracted MFCCs to create the DB Model  130   a  of the database. 
         [0024]    Individual Voice Modeler  140  is a module configured to build a mathematical model of each individual voice obtained from Audio Source  120   a.    
         [0025]    Voice Clusterer  150  is a module configured to cluster aka classify voices from Audio Source  120   a  into two or more groups, the Group  150   a  by characteristic inherent with each voice, including, but not limited to gender, pitch and speed. 
         [0026]    Group I-Vector  160  is a model configured to calculate a single i-vector for each Group  150   a.    
         [0027]    Target Voice Calculator  170  is a module configured to calculate the i-vector of the target voice, the Target i-Vector  170   a.    
         [0028]    Group Selector  175  is a module configured to select the closest Group  150   a  to the Target I-Vector  170   a , e.g. with the smallest Euclidean distance between the Target i-Vector  170   a  and the Group  150   a  or the highest probability score. 
         [0029]    Individual i-Vector  180  is a module configured to calculate the i-vectors of each Voice  180   a , the Voice  180   a  within the selected Group  150   a.    
         [0030]    Voice Selector  190  is a module configured to select the voice with the smallest Euclidean distance between the target i-Vector  170   a  and Voice  180   a.    
         [0031]      FIG. 2  illustrates a high flow diagram of the soundalike selection process. At step  210 , the soundalike system trains the database. At step  220 , the soundalike system builds mathematical models of each voice within the database. At step  230 , the soundalike system groups, i.e. creates clusters, of voices based on similarities between the voices e.g. pitch, speed, etc. step  240 , the soundalike system creates mathematical models of each cluster. At step  260 , the soundalike selects the cluster most likely to contain the soundalike voice. At step  270 , the soundalike system selects the voice from within the selected cluster that is closest to the target voice. 
         [0032]      FIG. 3  illustrates a flow diagram of training the Database  125 . At step  310 , the Input  120  received the Voice  120   b  from Database  125 . The Database  125  should contain enough Voice  120   b  to be statistically significant. Optimally Database  125  should contain at least 300 voices, each voice having spoken  300  sentences of 5 to 6 seconds duration per sentence. Thus Database  125  will have 300,000 to 340,000 seconds or approximately 55 to 66 hours of voice data. 
         [0033]    The Database  125  needs to be trained. Training a database means building a mathematical model to represent database. In speech synthesis, the ultimate result of training for soundalike is creating i-vectors for the cluster and speaker level. This is a final low dimension representation of a speaker. At Step  320 , the DB Trainer  130  divides the human speech into a plurality of frames, Frames  130   a , each Frame  130   a  being generally the length of a single phoneme or 30 milliseconds. At step  325 , DB Trainer  130  calculates N Mel Frequency Cepstral Coefficients, or MFCCS, for each Frame  130   a  which corresponds to the number of features extracted, i.e. the number of features in the target voice such as pitch, speed, etc., which will matched against the voices in the Database  125 . In the preferred embodiment, DB Trainer  130  calculates 42 MFCCs per Frame  130   a  over a sliding window equal which increments by ½ the length of Frame  130   a.    
         [0034]    At step  330 , the DB Trainer  130 , uses the extracted MFCCs from Database  125  to create UBM  130   b , a universal background model of the Database  125 . Creating a universal background model is within the scope of one skilled in the art of speech synthesis. The UBM  130   b  results in three matrices, the Weight  135   a , the Means  135   b  and the Variance  135   c.    
         [0035]    Subsequent to modeling the Database  125 , each Voice  120   b  must be modeled. At step  340 , the Individual Voice Modeler  140  builds a mathematical model for each Voice  120   b  using a Maximum Apriori Probability, or MAP, algorithm which combines the UBM  130   b  with the extracted MFCCs from each Voice  120   b . Building a mathematical model of a single voice using a Maximum Apriori Probability algorithm is within the ordinary scope of one skilled in the art of speech synthesis. 
         [0036]    In another embodiment, Individual Voice Modeler  140  creates a mathematical model of each voice directly using the universal background model. Building individual voice mathematical models using the universal background model algorithm is within the scope of one skilled in the art of speech synthesis. 
         [0037]      FIG. 4  illustrates a K-Means clustering. Applying a clustering algorithm is within the scope of one skilled in the art of speech synthesis. In the preferred embodiment, the clustering algorithm is a k-means algorithm. K-means stores k centroids that it uses to define clusters. A point is considered to be in a particular cluster if it is closer to that cluster&#39;s centroid than any other centroid. K-Means finds the best centroids by alternating between (1) assigning data points to clusters based on the current centroids (2) choosing centroids (points which are the center of a cluster) based on the current assignment of data points to clusters. 
         [0038]    There is no well-defined value for “k”, but experimentally, between 40 and 50 clusters is ideal for a database containing millions of voices. 
         [0039]      FIG. 4  illustrates a sample of k=2, i.e. two clusters (e.g. male and female voices). 
         [0040]    Once the number of clusters has been determined, the soundalike system builds a cluster model. A cluster model is a mathematical representation of each cluster within the selected database. A cluster model allows all of the voices within the cluster to be represented with a single mathematical model. 
         [0041]      FIG. 5  illustrates a flow diagram of the soundalike system creating a mathematical model for the database at the cluster level and calculating the i-vector of the target voice. At step  510  Group I-Vector  160  selects a single cluster of voices. At step  520 , Group I-Vector  160  selects the MFCCs from all of the voice within the selected cluster. At step  530 , the feature vectors, or MFCCs are combined together using any number of mathematical combinations. In the preferred embodiment, at step  530 , Group I-Vector  160  simply creates the matrix  160   a  by stacking the vectors, although other combinations such as summation, averages, means, etc. can be applied. A universal background model algorithm is applied to the Matrix  160   a . At step  540 , Group I-Vector  160  calculates the i-vector of the selected cluster. The result is the mathematical model of the selected cluster. Group I-Vector  160  repeats for each cluster in Database  125 . 
         [0042]    At step  550 , the Target Voice Selector  170  extracts the MFCCs of the target voice over a plurality of frames, each frame being approximately 20s, the length of a phoneme. In the preferred embodiment, the MFCC&#39;s are calculated over a sliding window equal in length to a single Frame  130   a    
         [0043]    At Step  560 , the Target i-Vector  165  is calculated by applying the universal background model to the MFCCs of the Voice  120   b . Calculating an i-Vector is within the scope of someone skilled in the art of speech synthesis. 
         [0044]      FIG. 6  illustrates a flow diagram of Group Selector  175  determining which group contains the soundalike voice. At step  610 , Group Selector  175  calculates the Euclidean distance between the i-vector of each group and the Target I-Vector  165 . At Step  620 , Group Selector  175  selects the Group with the lowest Euclidean distance to the Target I-Vector  165 . 
         [0045]    Once the Group  175   a  has been selected, the i-vectors of each individual voice must be calculated. 
         [0046]    At step  630 , Individual I-Vector  180  selects the Voice  120   b  within Group  175   a . At step  640  Individual I-Vector  180  calculates the i-vector of each Voice  120   b.    
         [0047]    At step  650 , Voice Selector  190  compares the I-Vector of each voice in Group  175   a  with the Target I-Vector  165  and closest I-vector as the soundalike voice. In the preferred embodiment of the invention, the soundalike system selects the Voice  120   b  with the smallest Euclidean distance to the target voice as the soundalike voice.