Abstract:
A pronunciation training system and methods are provided as a series of programmed routines stored on an item of removable storage media, and select information generated by a speech analysis engine to compute and display graphical representations of metrics useful to a student. The student selects from among a plurality of pre-recorded utterances spoken by a native speaker, and the student then records his or her pronunciation of the utterance. The software computes and displays graphical metrics for the native speaker&#39;s utterance and the student&#39;s utterance, in any of a variety of formats, on a side-by-side basis. The system also permits the student to repeat selected phrases and to monitor improvement by similarity between the graphical metrics.

Description:
FIELD OF THE INVENTION 
     The present invention relates to apparatus and methods for providing language-independent suprasegmental analysis and audio-visual feedback of prosodic features of a user&#39;s pronunciation. 
     BACKGROUND OF THE INVENTION 
     The increasing globalization of world economies makes it essential for individuals to be able to communicate successfully in languages other than their own. For individuals to be effective communicators in a new language, it is essential to learn proper pronunciation. Intonation, stress and rhythm are key prosodic features of effective communication and are critical for comprehension. Thus, there is a need for effective pronunciation teaching aids. 
     Pronunciation training, however, generally is perceived as difficult because it is often hard for a student to pick up the peculiarities of pronunciation of a foreign language. Often, it is difficult for the student even to recognize his or her own mistakes. Moreover, it is often difficult to train teachers to detect errors in pronunciation and therefore provide beneficial feedback to students. 
     Previously known language training systems generally may be grouped into two categories: (1) systems based on speech analysis (i.e., analysis of how an utterance is pronounced); and (2) systems based on speech recognition (i.e., recognition of what is said). Some systems use a combined approach, in which the speech is partially speech analyzed and partially recognized. 
     Commercially available systems in the speech analysis category are: The Speech Viewer, available from IBM Corporation, White Plains, N.Y.; VisiPitch, available from Kay Corporation, Lincoln Park, N.J.; and newer versions of Language Now, available from Transparent Language, Inc., Hollis, N.H. All of these systems extract and visualize the pitch of an utterance. An overview of methods for extracting pitch is provided, for example, at pages 197-208 of Parsons,  Voice and Speech Processing,  McGraw-Hill Book Company (1987). 
     A drawback common to all of the foregoing speech analysis methods is that they extract pitch independently of its relevancy to intonation pattern. Thus, for example, such systems extract pitch even for vocal noises. These previously known systems therefore do not address crucial prosodic parameters of speech, such as rhythm, stress and syllabic structure. 
     Commercially available systems in the speech recognition category are those offered by: The Learning Company, Knoxville, Tenn.; Transparent Language, Inc., Hollis N.H.; Syracuse Language Systems, Inc., Syracuse, N.Y.; and IMSI, San Rafael, Calif. In addition, several companies offer speech recognition engines, including: Dragon Systems, Newton, Mass.; IBM Corporation, White Plains, N.Y.; and Lernout &amp; Hauspie, Brussels, Belgium. 
     Most previously known language training systems present a sentence for a student to pronounce, record the student&#39;s utterance, and then calculate the distance between the student&#39;s utterance and one of a generalized native speaker. The calculated distance is presented to the student in the form of a indicator on a gauge and/or a graphical comparison of a waveform of the student&#39;s utterance to the waveform for a native speaker. 
     A disadvantage common to all of these previously known language training systems is that the grading of the student&#39;s utterance is arbitrary and non-specific. In particular, use of just a single parameter—the distance between the student&#39;s and native speaker&#39;s utterances—provides little useful information. This is because a speech signal represents an intrinsically multi-parametric system, and this richness is not quantifiable using the distance method alone. Additionally, the visual feedback of providing a graphical comparison of the student&#39;s unanalyzed waveform provides little useful information. Finally, all of the foregoing systems are language dependent. 
     Indeed, the literature relating to pronunciation training has recognized the shortcomings of available computer training systems for some time. See, e.g., D. M. Chun, “Signal Analysis Software for Teaching Discourse Intonation,”  Lang. Learning  &amp;  Tech.,  2(1):61-77 (1998); H. Norman, “Speech Recognition: Considerations for use in Language Learning,” EuroCALL &#39;98; and T. Van Els and K. de Bot, “The Role of Intonation in Foreign Accent,”  Modern Lang. J.,  71:147-155 (1987). 
     U.S. Pat. No. 5,799,276 describes a knowledge-based speech recognition system for translating an input speech signal to text. The system described in that patent captures an input speech signal, segments it based on the detection of pitch period, and generates a series of hypothesized acoustic feature vectors that characterizes the signal in terms of primary acoustic events, detectable vowel sounds and other acoustic features. A largely speaker-independent dictionary, based upon the application of phonological and phonetic/acoustic rules, is used to generate acoustic event transcriptions against which the series of hypothesized acoustic feature vectors are compared to select word choices. Local and global syntactic analysis of the word choices is provided to enhance the recognition capability of the system. 
     In view of the foregoing, it would be desirable to provide a voice and pronunciation training system that gives more meaningful audio-visual feedback than provided by previously known systems, by providing extensive audio-visual feedback pertinent to prosodic training. 
     It also would be desirable to provide a voice and pronunciation training system that provides easy-to-understand visualization of intonation, stress and rhythm patterns, visualizes syllabic structure of an utterance, and enables a user to pinpoint his or her pronunciation errors. 
     It further would be desirable to provide a voice and pronunciation training system that is curriculum independent and may be easily customized for different curricula. 
     It still further would be desirable to provide a voice and pronunciation training system that is language independent, thereby enabling a student to practice intonation, stress and rhythm patterns of a foreign language using his or her native language or free forms like “ta-ta-ta”. 
     It also would be desirable to provide a voice and pronunciation training system that enables suprasegmental analysis and visual feedback of an utterance for deaf and semi-deaf speakers who require visual feedback during speech training to compensate for hearing imparity, and for use by speech pathologists during their work with patients. 
     SUMMARY OF THE INVENTION 
     In view of the foregoing, it is an object of this invention to provide a voice and pronunciation training system that gives more meaningful audio-visual feedback than provided by previously known systems, by providing extensive audio-visual feedback pertinent to prosodic training. 
     It is also an object of this invention to provide a voice and pronunciation training system that provides easy-to-understand visualization of intonation, stress and rhythm patterns, visualizes syllabic structure of an utterance, and enables a user to pinpoint his or her pronunciation errors. 
     It further is an object of the present invention to provide a voice and pronunciation training system that is curriculum independent and may be easily customized for different curricula. 
     It is another object of this invention to provide a voice and pronunciation training system that is language independent, thereby enabling a student to practice intonation, stress and rhythm patterns of a foreign language using his or her native language or free forms like “ta-ta-ta”. 
     It is a still further object of the present invention to provide a voice and pronunciation training system that enables suprasegmental analysis and visual feedback of an utterance for deaf and semi-deaf speakers who require visual feedback during speech training to compensate for hearing imparity, and for use by speech pathologists during their work with patients. 
     These and other objects of the present invention are accomplished by providing a pronunciation training system and methods that analyze a user&#39;s utterance, extracts prosodic features and presents metrics corresponding to those features to a user in an easy-to-understand manner that is language and training curriculum independent. The pronunciation training system and methods are provided as a series of programmed routines stored on an item of removable storage media, such as an optical disk or high capacity floppy disk. 
     In particular, the student selects from among a plurality of pre-recorded utterances spoken by a native speaker to practice, and selects a graphical presentation mode. The student then records his or her pronunciation of the utterance. A speech analysis engine, such as described in U.S. Pat. No. 5,799,276 is used to analyze a user&#39;s utterance to extract prosodic features. The system then computes metrics for the student&#39;s utterance, and displays those metrics for the student&#39;s utterance and the native speaker&#39;s utterance on a side-by-side basis for the selected graphical presentation mode. The system also permits the student to repeat selected phrases and to monitor improvement by similarity between the graphical metrics. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     Further features of the invention, its nature and various advantages will be more apparent from the accompanying drawings and the following detailed description of the preferred embodiments, in which: 
     FIG. 1 depicts an illustrative computer system for implementing the software system of the present invention; 
     FIG. 2 depicts use of a system constructed in accordance with the principles of the present invention; 
     FIG. 3 depicts use of the system of the present invention to practice pronunciation of a particular utterance; 
     FIG. 4 illustrates a process of analyzing a student&#39;s utterance to extract prosodic features that are to be displayed as visual feedback to the student; 
     FIGS. 5A and 5B are illustrative examples of navigation screens of a sample curriculum in American English in accordance with a preferred embodiment of the present invention; 
     FIG. 6 is an example of a practice window in which no visualization mode has been selected; 
     FIG. 7 is an example of a practice window in which an Intonation Mode of visualization has been selected; 
     FIG. 8 is an example of a practice window in which a Stress/Rhythm Mode of visualization has been selected; 
     FIG. 9 is an example of a practice window in which a Waveform Mode of visualization has been selected; and 
     FIGS. 10A to  10 C are examples of syllable-by-syllable highlighting during playback in Intonation Mode. 
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     The present invention relates generally to a system and methods for voice training and accent reduction via audio-visual feedback of the prosodic features of speech. It will of course be understood by those skilled in the relevant arts that these exemplary embodiments of the invention in no way limit the intended use of the pronunciation tutor for accent improvement, and that the present invention may be used for other than accent improvement purposes, such as voice and speech training. In addition, while the present invention employs the knowledge-based speech recognition system described in U.S. Pat. No. 5,799,276, which is incorporated herein by reference, for a first stage of the speech analysis, other speech recognition or analysis systems also may be used. 
     Referring to FIG. 1, exemplary apparatus  20  suitable for use in implementing the present invention and methods is described. Apparatus  20  includes main processor  21  (and its associated RAM) and one or more data storage devices  22  (e.g., a hard disk, CD-ROM or DVD drive), and may comprise, for example, a multi-media personal computer, either portable or desk-top. Apparatus  20  further includes keyboard  23 , pointing device  24  (e.g., mouse or track ball), display screen  25 , microphone  26  and sound card and speakers  27 . As is conventional, pointing device  24  has means for positioning the cursor of display  25 , and one or more command keys for operating upon icons targeted by the cursor. Computer  20  preferably operates using a Windows operating environment, such as Windows 95®, Windows 98® or Windows NT 4.0® or higher, available from Microsoft Corporation, Redmond, Wash. The system of the present invention may be readily adapted to other computer operating systems. 
     In accordance with the present invention, a removable item of storage media  28 , illustratively a CD-ROM, is provided that may be loaded into a suitable data storage device  22  configured to read such media. The removable item of storage media stores a pronunciation and voice training system, constructed in accordance with the present invention, as a series of programmed routines. As is conventional, the software may be executed by the computer to enable the computer user (hereinafter referred to as a “student”) to practice and visualize pronunciation of a series of utterances. The system of the present invention may be implemented in apparatus  20  using Visual Basic or C++ programming languages, as is conventional for Windows-based graphical interfaces, and as would be readily understood by one skilled in the art of Windows-based programming. 
     Referring now to FIG. 2, use of a system constructed in accordance with the principles of the present invention is described. At block  31 , a student navigates through the curriculum by first selecting a training category. A sample screen display showing exemplary choices is depicted in FIG.  5 A. The student selects an utterance to practice at block  32 , for example, by clicking pointing device  24  over that utterance. At block  33  the student practices pronunciation of the utterance and adjusts his or her pronunciation based on the audio-visual feedback received from the system. Once the student has completed his or her exercises with respect to the utterance chosen at block  33 , the student may return to block  31  to practice a different part of the curriculum. 
     With respect to FIG. 3, the process of practicing pronunciation of a particular utterance is described. At block  41 , the student chooses, from a collection of pre-recorded utterances of a native speaker, an utterance to practice. In accordance with one aspect of the invention, the analysis is language independent, so that the system of the present invention may be readily adapted for use by students who study different languages. 
     Moreover, the system of the present invention is curriculum independent, so that the curriculum may be readily altered by substituting the type and complexity of pre-recorded native speaker utterances available to practice. Preferably, the pre-recorded native speaker utterances may be pre-processed using the techniques described hereinafter to extract the prosodic features, and the utterances and corresponding metrics may be stored as a replaceable module of the software program. 
     At block  42 , the student opens a practice window, an example of which is depicted in FIG.  5 . In accordance with another aspect of the present invention, at block  43 , the student chooses a visualization mode out of the three possible modes: Intonation Mode, Stress/Rhythm Mode and Waveform Mode. Intonation Mode displays a graphical representation of the pitch values on the vocal part of each syllable of the utterance. Stress/Rhythm Mode displays a graphical representation of each syllable of the intonation as steps, where the width of each step represents the duration of the corresponding syllable and the height of each step represents the energy in the vowel of the corresponding syllable. Waveform Mode displays a graphical representation of the speech waveform. Alternatively, the student may choose to receive audio feedback only. 
     At block  44 , the student optionally may choose to play a native speaker&#39;s pre-recorded utterance and study, depending on the visualization mode chosen, the intonation, stress/rhythm or waveform patterns for the utterance. The software also provides a graphical representation of the boundaries of the syllables of the utterance. 
     At block  45 , the student speaks the utterance into microphone  26 , and the sample utterance is digitized and recorded as described in the above-incorporated patent. The system then analyzes the sample utterance, at block  46 , as described in the above-incorporated patent and hereinbelow with respect to FIG. 4, and displays the metrics appropriate for the visualization mode selected. 
     At step  47 , the student plays the recording and displays the results of the analysis. Depending on the visualization mode chosen, Intonation Mode, Stress/Rhythm Mode or Waveform Mode, the appropriate metrics are displayed, along with an indication of the boundaries of the syllables of the utterance. The results of the analysis are displayed in a student&#39;s window, positioned beneath a graphical representation of the corresponding metric for the pre-recorded utterance, to permit ease of comparison (see, e.g., FIGS.  7 - 9 ). 
     Referring now to FIG. 4, the process of analyzing a student&#39;s utterance and extracting prosodic features to be displayed as visual feedback to the student is described. Blocks  51 - 54  may employ, for example, the process steps described in FIG. 4, and column 17, line 35 through column 33, line 37, of the above-incorporated U.S. Pat. No. 5,799,276. 
     In particular, the system starts the analysis of a student&#39;s recorded utterance at block  51  by dividing the utterance into segments that contain one or more continuously pronounced words and inter-word pauses. At block  52 , irrelevant segments at the beginning and end of the utterance, for example, attributable to such events as lip smacks, microphone clicks and other extraneous noises, are deleted. 
     At block  53 , the speech utterance is segmented into pitch synchronized frames by detecting the excitation points of vocal cords that constitute the beginnings of pitch synchronous frames. At block  54 , the speech utterance is segmented into noise, voice, burst and pause segments. At block  55 , voice intervals are segmented into vowels, semivowels, nasals, flaps and voiced closures. 
     For the purpose of syllable detection, vowels are defined as those segments that pass the following test: (1) the segment has formants; (2) the segment should not be a flap; (3) the vowel weight of the segment should be at least 60% of its nasal-glide weight. 
     Special considerations are required for parasitic vowels. For example, a segment that is not a vowel but has some features of a vowel may occasionally be mistaken for a vowel, such as with low energy, low periodical segments situated far from true vowels. These segments may be produced during hesitation at the beginning of the sentence or after a breath pause. If a suspicious vowel segment is further than 400 Msec from the closest consonant, has length less than 100 Msec and has energy lower than 10% of the neighboring vowels, the segment is deemed a parasitic vowel and is discarded. 
     At block  56 , syllables are constructed out of the segments detected at blocks  51 - 55 . By definition, a syllable is build around a vowel, and consists of one vowel and zero or more surrounding consonants. Thus, after vowel/non-vowel division is completed, each sequence of continuously pronounced words is regrouped according to the following rules: 
     1. Each vowel constitutes the core of one syllable; 
     2. If within a sequence there is only one vowel, then this sequence is one syllable that spans from the beginning to the end of the sequence; 
     3. If there is more than one vowel in the sequence, there may be contention as to which consonant between two vowels belongs to which syllable. That is, assuming that the sequence is . . . , v k , C 1 , . . . , c n , v k+1 , . . . , where v represents a vowel and c represents a consonant, it must be determined which of the consonants c i  belong to the syllable S k  generated by the vowel v k  and which of the consonants c i  belong to the syllable S k+1  generated by the vowel V k+1 . 
     In case (3) above, the following algorithm is applied to determine membership of a consonant. First, all consonants in the sequence that are located prior to the first vowel v 1  of the sequence belong to the first syllable s 1  formed by v 1 . Similarly, all consonants in the sequence that are located after the last vowel of the sequence belong to the last syllable of the sequence. For a consonant located between two vowels, if the energy of vowel v k  is greater than or equal to the energy of vowel v k+1 , then: 
     If n=1 then c 1 εS k . 
     If n&gt; 1  then c n εS k+1  while c 1 , . . . , c n−1 εS K . 
     The case when Energy(v k )&lt;Energy(v k+1 ) is treated analogously. 
     At block  57 , the system computes intonation, stress, rhythm and syllabic functions of the speech utterance. For each syllable the following parameters are calculated: 
     1. the energy of the vowel contained in the syllable; 
     2. the length of the syllable, in samples; 
     3. the coordinates of the beginning and the end of the vocal part of the syllable, in samples, where the vocal part of a syllable is defined as the maximal continuous vocal segment of the syllable containing the vowel of the syllable; and 
     4. the pitch values within the vocal part of the syllable. These values are calculated using pitch synchronous segmentation of the frames that are produced during the wave segmentation process described hereinabove with respect to block  51 . The pitch value is equal to the number of samples in one second divided by the length of a frame in samples. 
     The foregoing features are then graphically displayed to the student, as described hereinabove with respect to block  47  of FIG.  3 . 
     In a preferred implementation of the system of the present invention, the following information is presented to the student: in Intonation Mode, the pitch values on the vocal part of each syllable are presented as a graph; in Stress/Rhythm Mode, the syllables are presented as steps, where the width of a step represents the duration of the corresponding syllable and the height of a step represents the energy of the vowel in the corresponding syllable; and in Waveform Mode, the original speech signal is presented. In all visualization modes the syllable boundaries are presented during playback of the utterance. 
     With respect to FIGS. 5 through 10, preferred embodiments of a graphical user interface and methods of visualizing prosodic features are described. FIGS. 5A and 5B are exemplary screen displays of navigation screens for a sample curriculum. Boxes  64  and  69  show the current chapter, subchapter or exercise. Boxes  63  and  68  show the list of corresponding subchapters, exercises or utterances, depending on the student&#39;s current position within the curriculum. Buttons  61  and  66  are Exit buttons that return the student to the parent node in the navigation graph. Buttons  62  and  67  are Help buttons that may be used to access the Help file, which may include, for example, a detailed explanation of the curriculum. 
     FIG. 6 is an example screen of the practice window with no visualization mode chosen, for example, where a visualization mode has not yet been chosen, or the student chooses not to use visualization of the native speaker&#39;s pronunciation as a hint. The student may select one of the three visualization modes by selecting Intonation, Stress/Rhythm or Waveform modes in the block  79 . 
     Box  70  shows the text of the current utterance. Button  71  is an Exit button that returns the student to the parent node in the navigation graph. Button  72  is a Help button, as described hereinabove. Button  73  transfers the student to the previous utterance in the list of utterances in an exercise. Button  74  transfers the student to the next utterance in the list of utterances in an exercise. Button  75  is the Play button, which causes the computer to play back the pre-recorded native speaker&#39;s utterance. 
     Button  76  is the Record button, and signals the computer to initiate recording of students&#39; speech. The time interval allowed for the recording depends on the length of the utterance selected for practice, unless Stop button  77  is pressed sooner. When the student&#39;s speech is recorded, the analyzed utterance is played back using Play button  78 . 
     Referring to FIG. 7, an example screen is described that depicts a graphical representation of the utterance “He said what?” in the Intonation Mode of visualization of the present invention. Block  81  indicates that the Intonation Mode of visualization has been selected by the student. Box  82  displays the intonation metric of the selected pre-recorded native speaker&#39;s utterance. Box  83  displays the intonation metric of the student&#39;s recorded utterance. 
     FIG. 8 is an example screen depicting the graphical representation of the utterance “He said what?” in the Stress/Rhythm Mode of visualization of the present invention. Block  91  indicates that the Stress/Rhythm mode of visualization has been chosen. Box  92  displays the syllables of the selected pre-recorded native speaker&#39;s utterance as steps, where length L of a step represents the duration of the corresponding syllable and height H of a step represents the energy of the vowel contained in the corresponding syllable. Box  93  displays similar information for the syllables of the student&#39;s recorded utterance. 
     FIG. 9 is an example screen depicting the graphical representation of the utterance “He said what?” in the Waveform Mode of visualization of the present invention. Block  101  indicates that the Waveform mode of visualization has been chosen. Box  102  displays waveform samples of the pre-recorded native speaker&#39;s utterance. Box  103  displays waveform samples of the student&#39;s utterance. 
     Referring now to FIGS. 10A to  10 C, an example of syllable by syllable highlighting during playback in the Intonation mode is described, as implemented in a preferred embodiment of the system of the present invention. The selected pre-recorded utterance of the native speaker is “He said what?”. 
     FIG. 10A is a snapshot of the screen during playback of the first syllable of the utterance—“He”, where the portion of the intonation metric that corresponding to the syllable is highlighted (indicated by shaded area  111   a ). FIG. 10B is a snapshot of the screen during playback of the second syllable—“said”, where the portions of the intonation metric corresponding to the first two syllables are highlighted (shaded area  112   a ). FIG. 10C is a snapshot of the screen during playback of the third syllable “what”, where the portions of the intonation metric corresponding to the first three syllables are highlighted (shaded area  113   a ). In accordance with the principles of the present invention, the syllable by syllable highlighting illustrated in FIGS. 10A to  10 C also is preferably provided in the Stress/Rhythm Mode and Waveform Mode of visualization. 
     While the present invention is described as employing the knowledge-based speech recognition system described in the above-incorporated U.S. Pat. No. 5,799,276, it is to be understood that the analysis techniques and graphical presentation methods of the present invention may be readily adapted for use with other speech recognition or speech analysis apparatus and methods that provide suprasegmental analysis. 
     Moreover, modifications and additions may be made to adapt the system and methods of the present invention to a variety of applications where audio-visual feedback of the speech signal is desirable, as will be apparent to those skilled in the art based upon the foregoing disclosure. The foregoing embodiments are merely illustrations of the principles of this invention, and therefore various modifications can be made by those skilled in the art without departing from the scope and spirit of the invention.