Patent Publication Number: US-7912718-B1

Title: Method and system for enhancing a speech database

Description:
BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     The present invention relates to a feature for enhancing the speech database for use in a text-to-speech system. 
     2. Introduction 
     Recently, unit selection concatenative synthesis has become the most popular method of performing speech synthesis. Unit Selection differs from older types of synthesis by generally sounding more natural and spontaneous than formant synthesis or diphone-based concatenative synthesis. Unit selection synthesis typically scores higher than other methods in listener ratings of quality. Building a unit selection synthetic voice typically involves recording many hours of speech by a single speaker. Frequently the speaking style is constrained to be somewhat neutral, so that the synthesized voice can be used for general-purpose applications. 
     Despite its popularity, unit selection synthesis has a number of limitations. One is that once a voice is recorded, the variations of the voice are limited to the variations within the database. While it may be possible to make further recordings of a speaker, this process may not be practical and is also very expensive. 
     SUMMARY OF THE INVENTION 
     A system, method and computer readable medium that enhances a speech database for speech synthesis is disclosed. The method may include labeling audio files in a primary speech database and a secondary speech database, enhancing the primary speech database by placing the labeled audio files from the secondary speech database into the primary speech database, and storing the enhanced primary speech database for use in speech synthesis. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       In order to describe the manner in which the above-recited and other advantages and features of the invention can be obtained, a more particular description of the invention briefly described above will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings. Understanding that these drawings depict only typical embodiments of the invention and are not therefore to be considered to be limiting of its scope, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings in which: 
         FIG. 1  illustrates an exemplary diagram of a speech synthesis system in accordance with a possible embodiment of the invention; 
         FIG. 2  illustrates an exemplary block diagram of an exemplary speech synthesis system utilizing the speech database enhancement module in accordance with a possible embodiment of the invention; 
         FIG. 3  illustrates an exemplary block diagram of a processing device for implementing the speech database enhancement method in accordance with a possible embodiment of the invention; 
         FIG. 4  illustrates an exemplary flowchart illustrating one possible speech database enhancement method in accordance with one possible embodiment of the invention; 
         FIG. 5  illustrates an exemplary flowchart illustrating another possible speech database enhancement method in accordance with another possible embodiment of the invention; and 
         FIG. 6  illustrates an exemplary flowchart illustrating another possible speech database enhancement method in accordance with another possible embodiment of the invention. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The features and advantages of the invention may be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. These and other features of the present invention will become more fully apparent from the following description and appended claims, or may be learned by the practice of the invention as set forth herein. 
     Various embodiments of the invention are discussed in detail below. While specific implementations are discussed, it should be understood that this is done for illustration purposes only. A person skilled in the relevant art will recognize that other components and configurations may be used without parting from the spirit and scope of the invention. 
     The present invention comprises a variety of embodiments, such as a system, method, computer-readable medium, and other embodiments that relate to the basic concepts of the invention. 
     This invention concerns synthetic voices using unit selection concatenative synthesis where portions of the database audio recordings are modified for the purpose of producing a wider set of speech segments (e.g., syllables, phones, half-phones, diphones, triphones, phonemes, half-phonemes, demi-syllables, polyphones, etc.) than is contained in the original database of voice recordings. Since it is known that performing global signal modification for the purposes of speech synthesis significantly reduces perceived voice quality, the modifications that performed as described herein may be aperiodic portions of the signal that tend neither to cause concatenation discontinuities nor to convey much of the individual character or affect of the speaker. However, while it is generally easier to substitute aperiodic components than periodic components, periodic components can be substituted in accordance with the invention. While difficulty increases with increasing energy in the sound (such as with vowels), it is still possible to use the techniques described herein to substitute for almost all sounds, especially nasals, stops, fricatives, for example. In addition, if the two speakers have similar characteristics, then vowel substitution could also be more easily performed. 
     The speech database enhancement module  130  is potentially useful for applications where a voice may need to be extended in some way, for example to pronounce foreign words. As a specific example, the word “Bush” in Spanish would be strictly pronounced /b/ /u/ /s/ (SAMPA), since there is no /S/ in Spanish. However, in the U.S., “Bush” is often rendered by Spanish speakers as /b/ /u/ /S/. These loan phonemes typically are produced and understood by Spanish speakers, but are not used except in loan words. 
     There are languages, such as German and Spanish, where English, French, or Italian loan words are often used. There are also regions where there is a large population living in a linguistically distinct environment and frequently using and adapting foreign names. The desire would be to have the ability to synthesize such material accurately without having to resort to adding special recordings. Another problem may arise if the speaker is unable to pronounce the required “foreign” phones acceptably, thus rendering additional recordings impossible. 
     There are also instances in which the phonetic inventories differ between two dialects or regional accents of a language. In this case, expansion of the phonetic coverage of a synthetic voice created to speak one dialect to cover the other dialect is needed as well. 
     Thus, enhancing an existing database through phonetic expansion is a method to address the above issues. As an example, Spanish is used, and specifically on the phenomenon of “seseo,” one of the principal differences between European and Latin American Spanish. Seseo refers to the choice between /T/ or /s/ in the pronunciation of words. There is a general rule that in Peninsular (European) Spanish the orthographic symbols z and c (the latter followed by i or e) are pronounced as /T/. In Latin American varieties of Spanish these graphemes are always pronounced as /s/. Thus, for the word “gracias” (or “thanks”) the transcription would be /graTias/ in Peninsular Spanish or /grasias/ in Latin American Spanish. Seseo is one major distinction (but certainly not the only distinction) between Old and New World dialects of Spanish 
     Three methods are discussed in detail below to extend the phonetic coverage of unit selection speech: (1) by modifying parts of a speech database so that extra phones extracted from a secondary speech database can be added off line; (2) by extending the above methodology by using a speech representation model (e.g., harmonic plus noise model (HNM), etc.) in order to modify speech segments in the speech database; and (3) by combining recorded inventories from two speech databases so that at synthesis time selections can be made from either. While three methods are shown as examples, the invention may encompass modifications to the processes as described as well other methods that perform the function of enhancing a speech database. 
       FIG. 1  illustrates an exemplary diagram of a speech synthesis system  100  in accordance with a possible embodiment of the invention. In particular, the speech synthesis system  100  includes text-to-speech synthesizer  110 , primary speech database  120 , speech database enhancement module  130  and secondary speech database  140 . The speech synthesizer  110  represents any speech synthesizer known to one of skilled in the art which can perform the functions of the invention disclosed herein or the equivalence thereof. In its simplest form, the speech synthesizer  110  takes text input from a user in one or more of several forms, including keyboard entry, scanned in text, or audio, such as a foreign language which has been processed through a translation module, etc. The speech synthesizer  110  then converts the input text to a speech output using inputs from the primary speech database  120  which is enhanced by the speech database enhancement module  130 , as set forth in detail below. 
       FIG. 2  shows a more detailed exemplary block diagram of the text-to-speech synthesis system  100  of  FIG. 1 . The speech synthesizer  110  includes linguistic processor  210 , unit selector  220  and speech processor  230 . The unit selector  220  is connected to the primary speech database  120 . As stated in  FIG. 1 , the text-to-speech synthesis system  100  also includes the speech database enhancement module  130  and secondary speech database  140 . The primary speech database  120  may be any memory device internal or external to the speech synthesizer  110  and the speech database enhancement module  130 . The primary speech database  120  may contain raw speech in digital format, an index which lists speech segments (syllables, phones, half-phones, diphones, triphones, phonemes, half-phonemes, demi-syllables, polyphones, etc.) in ASCII, for example, along with their associated start times and end times as reference information, and derived linguistic information, such as stress, accent, parts-of-speech (POS), etc. 
     Text is input to the linguistic processor  210  where the input text is normalized, syntactically parsed, mapped into an appropriate string of speech segments, for example, and assigned a duration and intonation pattern. A string of speech segments, such as syllables, phones, half-phones, diphones, triphones, phonemes, half-phonemes, demi-syllables, polyphones, etc., for example, is then sent to unit selector  220 . The unit selector  220  selects candidates for requested speech segment sequence with speech segments from the primary speech database  120 . The unit selector  220  then outputs the “best” candidate sequence to the speech processor  230 . The speech processor  230  processes the candidate sequence into synthesized speech and outputs the speech to the user. 
       FIG. 3  illustrates an exemplary speech database enhancement module  130  which may implement one or more modules or functions shown in  FIGS. 1-4 . Thus, exemplary speech database enhancement module  130  may include may include a bus  310 , a processor  320 , a memory  330 , a read only memory (ROM)  340 , a storage device  350 , an input device  360 , an output device  370 , and a communication interface  380 . Bus  310  may permit communication among the components of the speech database enhancement module  130 . 
     Processor  320  may include at least one conventional processor or microprocessor that interprets and executes instructions. Memory  330  may be a random access memory (RAM) or another type of dynamic storage device that stores information and instructions for execution by processor  320 . Memory  330  may also store temporary variables or other intermediate information used during execution of instructions by processor  320 . ROM  340  may include a conventional ROM device or another type of static storage device that stores static information and instructions for processor  320 . Storage device  350  may include any type of media, such as, for example, magnetic or optical recording media and its corresponding drive. 
     Input device  360  may include one or more conventional mechanisms that permit a user to input information to the speech database enhancement module  130 , such as a keyboard, a mouse, a pen, a voice recognition device, etc. Output device  370  may include one or more conventional mechanisms that output information to the user, including a display, a printer, one or more speakers, or a medium, such as a memory, or a magnetic or optical disk and a corresponding disk drive. Communication interface  380  may include any transceiver-like mechanism that enables the speech database enhancement module  130  to communicate via a network. For example, communication interface  380  may include a modem, or an Ethernet interface for communicating via a local area network (LAN). Alternatively, communication interface  380  may include other mechanisms for communicating with other devices and/or systems via wired, wireless or optical connections. In some implementations of the network environment  100 , communication interface  380  may not be included in exemplary speech database enhancement module  130  when the speech database enhancement process is implemented completely within a single speech database enhancement module  130 . 
     The speech database enhancement module  130  may perform such functions in response to processor  320  by executing sequences of instructions contained in a computer-readable medium, such as, for example, memory  330 , a magnetic disk, or an optical disk. Such instructions may be read into memory  330  from another computer-readable medium, such as storage device  350 , or from a separate device via communication interface  380 . 
     The speech synthesis system  100  and the speech database enhancement module  130  illustrated in  FIG. 1  and the related discussion are intended to provide a brief, general description of a suitable computing environment in which the invention may be implemented. Although not required, the invention will be described, at least in part, in the general context of computer-executable instructions, such as program modules, being executed by the speech database enhancement module  130 , such as a general purpose computer. Generally, program modules include routine programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that other embodiments of the invention may be practiced in network computing environments with many types of computer system configurations, including personal computers, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. Embodiments may also be practiced in distributed computing environments where tasks are performed by local and remote processing devices that are linked (either by hardwired links, wireless links, or by a combination thereof) through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices. 
     For illustrative purposes, the speech database enhancement process will be described below in relation to the block diagrams shown in  FIGS. 1 ,  2  and  3 . 
       FIG. 4  is an exemplary flowchart illustrating some of the basic steps associated with a speech database enhancement process in accordance with a possible embodiment of the invention. In this process, waveform segments in the primary speech database  120  are directly substituted by others from the secondary speech database  140 . This segment substitution process may be performed offline. The process begins at step  4100  and continues to step  4200  where the speech database enhancement module  130  labels audio files in the primary speech database  120 . At step  4300 , the speech database enhancement module  130  identifies segments in the labeled audio files that have varying pronunciations based on language differences. Language differences may be a separate language, for example, such as English and Spanish, the result of dialect, geographic, or regional differences, such as Latin American Spanish and European Spanish, accent differences, national language differences, idiosyncratic speech differences, database coverage differences, etc. Database coverage differences may result from a lack or sparsity of certain speech units in a database. Idiosyncratic speech differences may concern the ability to imitate the voice of another individual. 
     Identification of segments to be replaced may be performed by locating obstruents and nasals, for example. The obstruents covers stops (b,d,g,p,t,k), affricates covers (ch,j), and fricatives covers (f,v,th,dh,s,z,sh,zh), for example 
     At step  4400 , the speech database enhancement module  130  identifies replacement segments in the secondary speech database  140 . At step  4500 , the speech database enhancement module  130  enhances the primary speech database  120  by substituting the identified secondary speech database  140  segments for the corresponding identified segments in the primary speech database  120 . At step  4600 , the speech database enhancement module  130  stores the enhanced primary speech database  120  for use in speech synthesis. The process goes to step  4700  and ends. 
     As an illustrative example of the  FIG. 4  process, the speech database enhancement module  130  may identify segments in the primary speech database  120  that could be substituted by a different fricative. For example, the speech database enhancement module  130  may identify the /s/ fricatives in the primary speech database  120  that in Peninsular Spanish would be pronounced as /T/. Because the unit boundaries in a unit selection database such as the primary speech database  120  are not always, or even necessarily, on phone boundaries, and the process may mark the precise boundaries of the fricatives or other language units of interest, independent of any labeling that exists in the primary speech database  120  for the purposes of unit selection synthesis. 
     Again, using fricatives as an example, the speech database enhancement module  130  can readily identify the /s/ in the primary speech database  120  and /T/ in the secondary speech database  140  in a majority of cases by relatively abrupt C-V (unvoiced-voiced) or V-C (voiced-unvoiced) transitions. The speech database enhancement module  130  may locate the relevant phone boundaries using a variant of the zero-crossing calculation or some other method known to one of skill in the art, for example. The speech database enhancement module  130  may treat other automatically-marked boundaries with more suspicion. In any event, the goal is for the speech database enhancement module  130  to establish reliable phone boundaries, both in the primary speech database  120  and in the secondary speech database  140 . 
     Once identified, the speech database enhancement module  130  may splice the new /T/ audio waveforms from the secondary speech database  140  into the primary speech database  120  in place of the original /s/ audio, with a smooth transition. With the new audio files and associated speech segment (e.g., syllables, phones, half-phones, diphones, triphones, phonemes, half-phonemes, demi-syllables, polyphones, etc.) labels, a complete voice was built in the normal fashion in the primary speech database  120  which may be stored and used for unit selection speech synthesis. 
       FIG. 5  is an exemplary flowchart illustrating some of the basic steps associated with a speech database enhancement process in accordance with another possible embodiment of the invention. The process begins at step  5100  and continues to step  5200  where the speech database enhancement module  130  labels audio files in the primary speech database  120 . At step  5300 , the speech database enhancement module  130  identifies segments in the labeled audio files that have varying pronunciations based on language differences as discussed above. 
     At step  5400 , the speech database enhancement module  130  modifies the identified segments in the primary speech database  120  using selected mappings. At step  5500 , the speech database enhancement module  130  enhances the primary speech database  120  by substituting the modified segments for the corresponding identified database segments in the primary speech database  120 . At step  5600 , the speech database enhancement module  130  stores the enhanced primary speech database  120  for use in speech synthesis. The process goes to step  5700  and ends. 
     As an illustrative example of the  FIG. 5  process, the speech database enhancement module  130  may use a speech representation model rather than the audio waveforms themselves, such as a harmonic plus noise model (HNM). In this process, the speech database enhancement module  130  may first convert the entire primary speech database  120  to HNM parameters. For each frame there is a noise component represented by a set of autoregression coefficients and a set of amplitudes and phases to represent the harmonic component. The speech database enhancement module  130  then modifies the HNM parameters. For example, the speech database enhancement module  130  may modify only the autoregression coefficients when a frame fell time-wise into one of the segments marked for change. In these cases, the modified autoregression coefficients were directly substituted for the originals in the primary speech database  120 . The speech database enhancement module  130  may then store the modified set of HNM parameters along with the associated phone labels in the primary speech database  120  for use in unit selection speech synthesis. Alternatively, the primary speech database  120  may be converted to HNM parameters, be modified as described above, and then converted back to a different (or third) speech database. 
       FIG. 6  is an exemplary flowchart illustrating some of the basic steps associated with a speech database enhancement process in accordance with another possible embodiment of the invention. This process involves the speech database enhancement module  130  combining the primary speech database and the secondary speech database  140  to get the benefits of both databases for speech synthesis. 
     The process begins at step  6100  and continues to step  6200  where the speech database enhancement module  130  labels audio files in the primary speech database  120  and secondary speech database  140 . At step  6300 , the speech database enhancement module  130  enhances the primary speech database  120  by placing the audio files from the secondary speech database  140  into the primary speech database  120 . At step  6400 , the speech database enhancement module  130  stores the enhanced primary speech database  120  for use in speech synthesis. The process goes to step  6500  and ends. 
     In this process, all the database audio files and associated label files for the two different voices may be combined. The speech database enhancement module  130  may choose to label the speech segments so that there will be no overlap of speech segments (phonetic symbols). Naturally, segments marked as silence may be excluded from this overlap-elimination process due to the fact that silence in one language sounds much like silence in another. Using these audio files and associated labels a single hybrid voice was built. 
     The speech database enhancement module  130  may label the primary speech database  120  with a labeling scheme distinct from the secondary speech database  140 . This process may provide for easier identification by the unit selector  220 . Alternatively, the speech database enhancement module  130  may label the primary speech database  120  with the same labeling scheme as the secondary speech database  140 . In that instance, the duplicate segments may be discarded or be allowed to remain in the primary speech database  130 . 
     As a result of the  FIG. 6  process, access to the voice can be controlled at the phoneme level, with the choice of phones determining whether one voice will be heard in English, or the other voice in Spanish. The speech database enhancement module  130  may substitute phones simply by specifying a different phone symbol for particular cases. For example, the speech database enhancement module  130  may specify a /T/ unit rather than a /s/ unit in appropriate instances. Note that in this case the speech database enhancement module  130  makes no attempt to refine whatever phoneme boundaries were defined in the original primary speech database  120  itself. Often these boundary alignments can be less accurate than desired for the purposes of unit substitution. 
     Embodiments within the scope of the present invention may also include computer-readable media for carrying or having computer-executable instructions or data structures stored thereon. Such computer-readable media can be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store desired program code means in the form of computer-executable instructions or data structures. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or combination thereof) to a computer, the computer properly views the connection as a computer-readable medium. Thus, any such connection is properly termed a computer-readable medium. Combinations of the above should also be included within the scope of the computer-readable media. 
     Computer-executable instructions include, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Computer-executable instructions also include program modules that are executed by computers in stand-alone or network environments. Generally, program modules include routines, programs, objects, components, and data structures, etc. that perform particular tasks or implement particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of the program code means for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps. 
     Although the above description may contain specific details, they should not be construed as limiting the claims in any way. Other configurations of the described embodiments of the invention are part of the scope of this invention. For example, the principles of the invention may be applied to each individual user where each user may individually deploy such a system. This enables each user to utilize the benefits of the invention even if some or all of the conferences the user is attending do not provide the functionality described herein. In other words, there may be multiple instances of the speech database enhancement module  130  in  FIGS. 1-3  each processing the content in various possible ways. It does not necessarily need to be one system used by all end users. Accordingly, the appended claims and their legal equivalents should only define the invention, rather than any specific examples given.