Source: https://patents.google.com/patent/US8972268B2/en
Timestamp: 2018-05-21 21:10:39
Document Index: 646988246

Matched Legal Cases: ['Application No. 61', 'Application No. 61', 'Application No. 61', 'Application No. 09732921', 'Application No. 2011', 'Application No. 10']

US8972268B2 - Enhanced speech-to-speech translation system and methods for adding a new word - Google Patents
Enhanced speech-to-speech translation system and methods for adding a new word Download PDF
US8972268B2
US8972268B2 US13008346 US201113008346A US8972268B2 US 8972268 B2 US8972268 B2 US 8972268B2 US 13008346 US13008346 US 13008346 US 201113008346 A US201113008346 A US 201113008346A US 8972268 B2 US8972268 B2 US 8972268B2
US13008346
US20110307241A1 (en )
A speech translation system and methods for cross-lingual communication that enable users to improve and modify content and usage of the system and easily abort or reset translation. The system includes a speech recognition module configured for accepting an utterance and adding a new word, a machine translation module, an interface configured to communicate the utterance and proposed translation, a correction module and an abort action unit that removes any hypotheses or partial hypotheses and terminates translation. The system also includes modules for storing favorites, changing language mode, automatically identifying language, providing language drills, viewing third party information relevant to conversation, among other things.
The present application is a continuation in part of U.S. patent application Ser. No. 12/689,042 filed on Jan. 18, 2010, which is a continuation in part of U.S. patent application Ser. No. 11/925,048. This application is also a continuation in part of U.S. patent application Ser. No. 12/424,311 filed on Apr. 15, 2009, which claims priority to U.S. Provisional Patent Application No. 61/045,079 filed on Apr. 15, 2008 and claims priority to U.S. Provisional Patent Application No. 61/092,581 filed on Aug. 28, 2008 and claims priority to U.S. Provisional Patent Application No. 61/093,898 filed on Sep. 3, 2008.
Automatic speech recognition (ASR) and machine translation (MT) technologies have matured to the point where it has become feasible to develop practical speech translation systems on laptops or mobile devices for limited and unlimited domains. Domain limited speech-to-speech systems, in particular, have been developed in the research field and in research laboratories for a variety of application domains, including tourism, medical deployment and for military applications. Such systems have been seen before in the works of A. Waibel, C. Fugen, “Spoken language translation” in Signal Processing Magazine, IEEE May 2008; 25(3):70-79; and Nguyen Bach, Matthias Eck, Paisarn Charoenpornsawat, Thilo Köhler, Sebastian Stüker, ThuyLinh Nguyen, Roger Hsiao, Alex Waibel, Stephan Vogel, Tanja Schultz and Alan W. Black, for examples. “The CMU TransTac 2007 eyes-free and hands-free two-way speech-to-speech translation system,” In Proc. of the IWSLT, Trento, Italy, Oct. 2007. They are limited, however, in that they operate with a limited vocabulary which is defined by the developers of the system in advance, and is determined by the application domain, and the location where it is envisioned the system will be used. Thus vocabularies and language usage are determined largely based on example scenarios and by data that is collected or presumed in such scenarios.
In various embodiments, the present invention solves the foregoing problems by providing a system and methods for updating the vocabulary of a speech translation system. A computer assisted method is provided for overriding the recognition or translation of an utterance input in a speech translation system for translating a first language into a second language. The method includes accepting an utterance spoken in the first language, adds the utterance to a first automatic speech recognition module of the first language, translates the utterance to a corresponding translation in the second language using a first machine translation module, generates a speech output for the translated utterance via a text to speech module, and associates a description with the utterance. The description contains text of the utterance, a pronunciation, a translation and a pronunciation of the translation. The system then prompts the user to verify the description, and updates the utterance and the user verified description in a first machine translation module associated with the first language. In the event of an error, such as a misspoken utterance, the method aborts translation to remove any hypotheses or partial hypotheses that may have been created, terminates production of translation; and resets the translation system to accept a new utterance.
In embodiments, the present invention is directed to a field maintainable class-based speech translation system for translating a first language into a second language. The translation system includes a speech recognition module of a first language, a first machine translation module, an interface, a correction module and an abort action unit. The speech recognition module accepts sound comprising an utterance in a first language, determines if it is a new utterance and associates a description with the new utterance. The first machine translation module is associated with the first language and comprises a first tagging module, a first translation model and a first language module. The description contains text of the utterance, a pronunciation, a translation and a pronunciation of the translation wherein the pronunciation and translation are generated via rule-based or statistical models. The interface is configured to output to a user the description of the new utterance. The correction module is configured to accept the user's verification or correction of the pronunciation and translation of the new utterance via user editable phonetic transcription, wherein the first machine translation module is configured to be updated with the new utterance and the description. The abort action unit is configured to abort processing of the utterance in the first speech recognition module, the first machine translation or both upon request from a user.
In various embodiments a method is provided for updating the vocabulary of a speech translation system for translating a first language into a second language including written and spoken words. The method includes adding a new word in the first language to a first recognition lexicon of the first language and associating a description with the new word, wherein the description contains a pronunciation, a translation a pronunciation of the translation and optionally the description also contains word class information. The new word and description are then updated in a first machine translation module associated with the first language. The first machine translation module contains a first tagging module, a first translation model and a first language module, and is configured to translate the new word to a corresponding translated word in the second language. Optionally, for bidirectional translation, the method additionally includes the steps of translating the translated word from the second language back into the new word of the first language, correlating the new word with a corresponding translated word of the second language and adding the translated word and its description to a second recognition lexicon of the second language. A second machine translation module associated with the second language is then updated with the translated word and the description. The second machine translation module contains a second tagging module, a second translation model and a second language module
In embodiments the method further comprises the further step of inputting the first word into a text-to-speech pronunciation lexicon associated with the first language, and inputting the second word into a text-to-speech pronunciation lexicon associated with the second language. The input signals may be of different modalities (eg. speech and nonverbal spelling, speech and verbal spelling, writing and speech, etc.) (referred to herein as “cross-modal”) or may be of the same modality (speech and respeaking, writing and rewriting, etc.).
An embodiment of the invention is directed to a field maintainable class-based speech-to-speech translation system embodied in a device for communicating between a first language and a second language. The system includes two speech recognition units, each configured for accepting sound comprising the spoken word of the first or second language and for producing text that corresponds to the spoken language, and two corresponding machine translation units, each configured to receive text from one of the speech recognition units and output a translation of the text into text of the other language. It also includes a user field customization module that enables the system to learn new words in cooperation with the user. The user field customization module is configured for accepting user selected input that comprises sounds or text corresponding to one or both of the languages and updates the machine translation units appropriately with the user selected input.
In an embodiment, four primary features equip the system to provide a field maintainable class-based speech-to-speech translation system. The first includes a speech translation framework that enables the addition of new words to the active system vocabulary implemented in the deployed system, or the switching between location or task specific vocabularies. This provides for dynamic addition of words to a speech recognition module without requiring the module to be re-started. The system uses multilingual system-dictionary and language independent word-classes across all system components in the speech-to-speech translation device, class-based machine-translation (phrase-based statistical MT, syntactic, example-based, etc,), multilingual word-class tagging during model training, based on combination of monolingual taggers, and word-class tagging in new language by way of alignment via parallel corpus from known tagged language. Second, a multimodal interactive interface enables non-experts to add new words to the system. Third, the system is designed to accommodate ASR and SMT model adaptation using multimodal feedback provided by the user. And fourth, the system has networking capability to enable sharing of corrections or words.
In another embodiment, a multimodal interactive interface enabling a user to add new words to a speech-to-speech translation device in the field and without technical expertise is disclosed. Examples include: (1) Methods to automatically classify class of word or word-phrase to be added to the system, and automatically generate pronunciations, and translation of the word; (2) Method for entering new words cross-modally by one or more of speaking, typing, spelling, handwriting, browsing, paraphrasing; (3) Multimodal feedback to help a linguistically untrained user determine if the phonetic pronunciation and translation is adequate. This includes: multiple textual forms of the written form (romanized form as well as written form in other language's script), multiple textual forms of the phonetic pronunciation (IPA pronunciation, local method of pronunciation, including Katakana for Japanese or Pingying for Chinese, as well as pseudo-phonetic spelling for English, Irish and suitable pronunciation notations) and acoustic representations via text-to-speech (TTS; i.e. does it sound right); (4) Method for setting language model and translation probabilities for new word; and (5) Boosting or discounting language model and translation probabilities for new learned word based on relevance to user activities, interests and history of use.
In another embodiment, an interne application that allows users to share corrections or new word additions done in the field across devices is disclosed. Examples include: (1) Methods to upload, download and edit models for use in speech-to-speech translation devices via the world-wide-web; (2) Methods to collate in-the-field new words additions and corrections across the entire community of users; and (3) Methods to upload, download and edit, location or task specific vocabularies for use in speech-to-speech translation devices.
The usability of speech translation systems is further enhanced by incorporating additional functionalities disclosed in further detail in the detailed description. For examples in various embodiments, the speech translation system and methods include one or more additional modules. These modules, for examples, provide capabilities for the user to add phrases to the system via explicit input or correction, store favorites, automatically identify language, and create and utilize language drills. In examples, modules enable the system to change and/or adapt the language mode either automatically or manually by the user for leveraging discourse, environment and/or user context within the system. In examples, a module enables the system to bleep out rude words. In examples, the system connects to or is integrated with an external device for speech translation of incoming audio. In other examples the system provides continuous translation of incoming audio, automatically prompts back-channels, enables speech translation over the internet, and/or extracts information for targeted advertising/hotel information. These modules may be used with the speech translation system or methods alone or in combination with any of the other modules depending upon the anticipated or desired use of the speech translation system.
FIG. 1 illustrates a block diagram overview of an example of a field maintainable speech-to-speech translation system according to the present invention. In this example the system operates between two languages La and Lb. This is the typical implementation of a speech-to-speech dialog system involving speech-to-speech translation in both directions, from La to Lb and from Lb to La. However, the bi-directionality of this configuration is not a prerequisite for the present disclosure. A uni-directional system from La to Lb, or a multi-directional system involving several languages L1 . . . Ln could equally benefit from the present invention. The system has two ASR modules 2 and 9, that recognize speech for La and Lb, respectively, and produce text corresponding to La and Lb, respectively using acoustic model 18, ASR class-based language model 19 and a recognition lexicon model 20 (shown in FIG. 3). In this example, we used the “Ninja” speech recognizer system developed at Mobile Technologies, LLC. Other types of ASR modules which may be used include speech recognizers developed by IBM Corporation, SRI, BBN or at Cambridge or Aachen.
In an embodiment, an information extraction module, 12 a, will be present which extracts information from recent system logs 14 e (for language La) and 14 f (for language Lb), and presents targeted advertisement and supportive information via a graphical user interface displayed on the screen of the device 13. Relevant information will be obtained from the internet or a local knowledgebase.
In an embodiment, a language learning module, 12 b, will be present which extracts information from recent system logs (14 e and 14 f) to subsequently be used for language learning. Drills are automatically generated based on recent system usage by the user, additionally the user can choose any sentence output from the system to add to his/her active learning drill (step 34 c).
Thereafter, if the user is dissatisfied with the generated translation, the user may intervene during the speech-to-speech translation process in any of steps from 27 to 33 or after process has completed. This invokes the Correction and Repair Module 11 at (step 35). The Correction and Repair Module (Module 11) records and logs any corrections the user may make, which can be later used to update ASR modules 2 and 9 and MT modules 3 and 8 as described in detail further below in this document. If the correction contains a new vocabulary item (step 36), or if the user enters the field customization mode to explicitly add a new word to the system in step 15 c, or if a new word is automatically detected in the input audio using confidence measures or new word models, such as the method described in Thomas Schaaf, “Detection of OOV words using generalized word models and a semantic class language model,” in Proc. of Eurospeech, 2001 in step 15 d; the User Field Customization Module (Module 12) is invoked.
In addition to the consecutive translation mode where the user holds down a push-to-talk button (step 15 b) and speaks only a single utterance per-term, in an embodiment of the system a simultaneous translation mode will be present. In this mode no button push is required but the system continuously recognizes and translates all speech present on both microphone inputs (FIG. 1, items 1 and 10). Continuously recognition and simultaneous translation is shown by steps 15 e and 34 a.
In addition to the speech translation modes the user can exit the main system to enter either the “field customization” mode (step 15 c), the “translations favorites” mode (step 15 f), or the “language learning” mode (step 15 g).
During use any sentence-pair outputted by the system can be added to the users “translations favorites” list (step 34 b).
The User Field Customization Module (Module 12) provides a multimodal interface to enable users to add new words to the active system vocabulary. When a new word or phrase is added by a user the ASR, MT and TTS models (items 17, 21 and 33 a) are updated as required.
In the present invention, we introduce an abort action into the correction module. The abort action instantaneously aborts the speech-translation processes. It removes any hypotheses or partial hypotheses that may have already been output and resets the system to accept a new input. The abort action can be initiated by the user either depressing the push-to-talk button via the graphical user interface again (thus reinitiating recording for more input without waiting before the old one completes processing) or by shaking the phone to stop all processing. Output from internal accelerometers [14 a] embedded in the hardware device or from an embedded camera [14 b] that measures rapid shaking movement are used to determine if the device is being shaken by the user. The determination of the wrist shake by the user is then used to abort all processing that may be in progress and clear's the device's screen. The abort action can also be confirmed acoustically to the user with a crumbling paper noise, or other sound icon that confirms acoustically during field use that the speech translation process has been aborted. The simplicity of a shake of a wrist as well as the accompanying methods to confirm the abort by way of an acoustic sound icon, noise, beep, keyword or signal, provides simple fast, effective, unambiguous and intuitive signaling for both dialog partners that correction has taken place.
After the user indicates that he/she wishes to add a new word to the system vocabulary (step 50), the system first compares the word to entries in a large background recognition lexicon (item 50 d), as listed in FIG. 5, step 50 a. If the word is present in this lexicon then the listed pronunciations and matched n-gram entries from a large background speech recognition language model (item 50 e) are included into the active ASR models (item 17). This step is shown in FIG. 5, step 50 b. If the word is not present in the background recognition lexicon (item 50 d) the system next compares the word to a large external dictionary, which is either contained locally on the device, or is a dictionary service that can be accessed via the Internet, or is a combination of both. The external dictionary consists of entries of word translation pairs. Each entry contains pronunciation and word-class information which enables the new word to be easily added to the active system vocabulary. Each entry also contains a description of each word-pair in both languages. This will allow the user to select the appropriate translation of the word, even if they have no knowledge of the target language. If the new word is contained within the external dictionary (step 51), the system displays a list of alternative translations of the word with a description of each (step 52). If the user selects one of the predefined translations from the dictionary (step 53), then they can verify the pronunciation and other information provided by the dictionary (step 53 a), and the edit this if necessary. The phonetic pronunciation can be corrected using any suitable notation, including; IPA pronunciation, a local method of pronunciation notation, including Katakana for Japanese or Pingying for Chinese, a pseudo-phonetic spelling for English, or any other suitable phonetic notation. Since a user may not know how to write the correct phonetic spelling for a word, alternative pseudo-phonetic transcription are used in combination with synthesis to sound out the new word aloud. In this way, the user may type a name or new word according to his/her orthographic conventions, while the internal letter to sound conversion routines provide the conversion into phonetic notation based on each spelling. The word can then be played back according to the phonetic string, and if not satisfactory the user can iterate. Finally, when a satisfactory reading is achieved, the new word is then added to the active system vocabulary.
When the new word entered by the user is not found in the external dictionary, the system will automatically generate the information required to register the word into the active system vocabulary, and wll verify this information with the user. First, the class of the new word is estimated via a tagging model (FIG. 3, model 22) using the surrounding word context if it is available (step 54). Next, the pronunciation and translation of the new word are automatically generated via either rule-based or statistical models (step 55). The resulting information is then shown to the user via a multimodal interface (step 58). The system prompts the user to verify (step 58) or correct (step 57) the automatically generated translation or pronunciation. Finally, after the user has verified this information, the new word is added to the active system vocabulary (steps 59, 59 a, 59 b). To dynamically add a new word (specifically, “word+pronunciation+word class”) to the ASR vocabularies (59), the recognition lexicon 20 (which is typically stored as a tree-structure, within ASR Modules 2 or 9) is searched and then updated to include the new word. This enables the new word to be added to the recognition vocabulary dynamically, and it can thus be recognized, immediately, if spoken in the following utterance. The ASR system does not need to be re-initialized or re-started as in prior systems.
Such observations and relevance statistics are collected based on the user's observed location, history or activity, and/or alternatively by observing the occurrence of the system's new word in a large background language resource such as the interne. Such statistics may be collected monolingually, in a data-rich language and applied to the translation dictionary and translation language model.
One embodiment of class-based machine translation is class-based statistical machine translation, in which a foreign language sentence fJ 1=f1, f2, fJ is translated into another language eI 1=e1, e2, . . . , eI by searching for the hypothesis ^eI 1 with maximum likelihood, given:
^e I 1=argmax P(e I 1 |f J 1)=argmax P(f J 1 |e I 1)·P(e I 1)
To translate an input sentence the method illustrated in FIG. 11 is applied. First, the input sentence is normalized (step 105) and tagged (step 106) using a similar procedure as that applied to the training corpora. The input sentence is tagged using a monolingual tagger (FIG. 3, model 22). Next, the input sentence is decoded using class-based MT models (FIG. 3, models 23 and 74). For class-based statistical machine translation decoding is performed using the same procedure used in standard statistical machine translation, However, phrase-pairs are matched at the class-level, rather than the word, as shown in the example below.
@PLACE.city{
□□□@TIME{4:30}
λa P(Ta,Sa)·λb P(Tb,Sb) where, Oi(Ta)−Oi(Tb) for 1≦i≦M
In an embodiment, in the case where no manually annotated corpora is available for a specific language, labels can be generated by projecting labels from a first language where labels are known, across the sentence-pairs in the training corpora to the non-annotated language. One approach to do this is described in D. Yarowsky, G. Ngai and R. Wicentowski, “Inducting Multilingual Text Analysis Tools via Robust Projection across Aligned Corpora,” In Proc. HLT, pages 161-168, 2001 (“Yarowsky01”).
Monolingual and Bilingual Tagging Accuracy on Meld-Out Training Set
For the baseline system phrase-based translation models were trained using the Moses toolkit such as described in Koehn05 and GIZA++ (such as that used by Franz Josef Och, Hermann Ney. “A Systematic Comparison of Various Statistical Alignment Models”, Computational Linguistics, volume 29, number 1, pp. 19-51 Mar. 2003). 3-gram language models were trained using the SRILM toolkit of A. Stolcke “SRILM—an extensible language modeling toolkit”, In Proc. of ICSLP, pp. 901-904, 2002. Decoding was performed using our PanDoRA decoder. The decoder is described in Ying Zhang, Stephan Vogel, “PanDoRA: A Large-scale Two-way Statistical Machine Translation System for Hand-held Devices,” In the Proceedings of MT Summit XI, Copenhagen, Denmark, Sep. 10-14 2007. Systems were created for both translation directions J→E (Japanese to English) and E→J (English to Japanese) using the training set described in Table 1. The data used to train the target language models were limited to this corpora. The translation quality of the baseline system was evaluated on a test-set of 600 sentences. One reference was used during evaluation. The BLEU-score for the J→E and E→J systems were 0.4381 and 0.3947, respectively. BLEU-score is described in Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu “BLEU: a Method for Automatic Evaluation of Machine Translation,” In Proc. Association for Computational Linguistics, pp. 311-318, 2002. Translation quality using three different tagging schemes was evaluated:
Translation Quality (BLEU [Papineni02])
Frequently, users may say the same phrase or sentence repeatedly in a field situation. To eliminate the need to re-speak the same phrases and sentences over and over again, embodiments of the invention provide a speech translation favorites module, which stores frequently used phrases for rapid play-back. This favorites modules differs from a plain list or phrase book in one most important aspect: it gets filled and built by the speech translation device, and thus, does not require a bilingual speaker to be on hand to know the correct translation to a term or phrase. The favorites module is programmed to enable a user to perform the following functions: Copy bilingual sentence-pair from the speech translation interaction window to a favorites list; provide editing capability to the newly copied bilingual sentence-pair, so a user can modify both input and output string; and provide the ability to play back the target language side of the added sentence pair by applying the synthesis module. With the favorites list in place, a user can simply play back various accumulated phrases from the favorites list, without speaking them first, invoking them by clicking on the selected favorite or by way of a voice command or key phrase. This saves time in field situations. At the same time the favorites list provides the full flexibility of a full two-way speech-to-speech translator since it does not require a linguistically trained expert knowledgeable of both languages to build such a list. A customized list can be built by the user in the field and on the fly.
In addition to the basic speech translation system shown in FIG. 1 an embodiment of this system will add additional functionality to allow users to playback predefined phrases via TTS (FIG. 1, modules 3 and 7). An example graphical user interface for this functionality is shown in FIG. 16. While using the speech translation system the users can select any output generated by the system and add this to their favor list (step 34 b). The system will check if the sentence-pair already exists in the “Translation Favorites” list (step 109) and if not will append it (step 109 a). A possible graphical user interface for this phrase-addition step is shown in FIG. 13. In this example the user can press a graphical button (item 127) to add the current bilingual sentence-pair to the user's “Translation Favorites” list. Instead of performing speech translation, the user can enter the “Translations Favorites” mode (step 15 f). In this mode (detailed in FIG. 13) The system allows users to select a sentence-pair for playback (steps 110, 111), edit sentence-pairs currently in the list (steps 112, 113), delete a sentence-pair from the list (steps 114, 115) or append a sentence-pair to the list (steps 116, 117) by typing in the required information.
The basic speech translation system shown in FIG. 1 performs speech recognition (modules 2 and 9) using a single set of ASR models per language, machine translation (modules 3 and 8) using a single set of MT models per language-pair, and speech generation (modules 3 and 7) using a single set of TTS models per language. In one embodiment of this system, multiple language mode-dependent models will be available to the system rather than a single set of models. Specific models (ASR, MT and TTS) are developed for specific modes of communication including formal/colloquial/slang/rude, authoritative/submissive, and standard/dialectal language. Users either explicitly specify which mode they wish to use via the graphical user interface or the language mode can be estimated in the User Field Customization Module (item 12), based on device location, input speech or discourse history. Different language modes will generate different translation and speech output depending on the communication-style they are tuned towards, and will constrain the speech recognition components to the type of language used in that communication-style. In another embodiment, different language modes will be realized by simply filtering the output of the speech recognition (modules 2 and 9) and machine translation components (modules 3 and 8). For example while in formal mode, slang or rude phrases will be filtered from the ASR and MT output, be replaced with a star symbol (*) in the display of the graphical user interface, and be replaced with an audible beep during TTS (modules 3 and 7). The User Field Customization Module (item 12) will contain the specific lists of words to be filtered for each language mode.
The basic speech translation system shown in FIG. 1 operates over two languages, language La and language Lb. For consecutive translation the user holds down a push-to-talk button (FIG. 3, step 15 b) to start the speech recognition process (FIG. 3, step 27). The identity of the button pushed will determine if recognition is performed in language La (using FIG. 1, module 2) the primary users language, or in language Lb (using FIG. 1, module 9) the language of the dialog participant. In addition to this approach, an embodiment of the system will perform automatic identification of the input language (step 27 a) to select the appropriate ASR module to use for recognition (step 27 b). In another embodiment, language identification (step 27 a) is used to select the appropriate ASR module where modules exist for more than two languages. In another embodiment, it is also applied during simultaneous translation.
In addition to the basic speech translation system shown in FIG. 1 an embodiment of this system will add additional functionality to allow users to learn language Lb as they use the system in the field. The language learning module (item 12 a) observes sentences a user has uttered over a period of time (items 14 e, 14 f) and builds a profile of common language usage statistics. Statistics include typical syntactic constructs and vocabulary usage. When the user enters the language learning mode (step 15 g), the system will consolidate statistics from recent system logs (step 118) and use this information to construct vocabulary drills based on word usage by the user and related words based on topic or usage as determined by word frequency or semantic word clustering (step 119). A similar approach will be used to generate syntax drills (step 120). The drills will be presented to the user (step 121), and if performance is not satisfactory (step 122) the user will perform the drill again. In parallel to the automatic construction of language learning drills, the user is also provided with direct control over his/her language learning drills. Each sentence spoken in the speech translator can also be directly copied to the learning module (step 34 c), so that its words and expressions appear in the subsequent learning drills. When sentences are added to the active learning drill, the system first checks if the drill contains the sentence (step 118 a) and if it does not before appends it (step 118 b).
The basic speech translation mode operates using speech input from microphones (items 1 and 10) and loudspeakers (items 5 and 6) located on the device. In addition to this mode, speech translation can also be performed over a telephony network. In this case audio for language Lb is transmitted over the telephone network and the microphone (item 10) and loudspeaker (item 5) for language Lb will be physically located on a recipient's telephony handset. When operated over a telephony network speech recognition (modules 2, 9) and machine translation (modules 3, 8) can be carried out in two different manners. In consecutive translation the user holds a push-to-talk button down (step 15 b) to start recognizing an utterance (step 27). The utterance is then recognized and translated using the steps shown in FIG. 3. In simultaneous translation (steps 15 d, 34 a) no button push is required but the system continuously recognizes and translates all speech present on both microphone inputs (items 1 and 10). Speech translation output from (step 33) is then provided either acoustically, overlaying or in place of the original speaker's speech, or visually, by displaying text on a user's telephone device.
Speech Recognizers and Translators operating on a smart phone can also provide information as to a user's speech content. We propose to expand the speech recognizer and translator to extract topical information from two conversants' speech. Such information is then used to seek relevant related information on the internet. Such information is then presented to the user. There are multiple uses of such conversation enabled information extraction. It could be used to provide more targeted advertising (perhaps in return for cheaper calling rates). It could also be used to provide the user with helpful supporting information (for example, calling up flight schedules, hotel availabilities & rates, recalling a person's contact details, etc.) when a conversation mentions certain topics, people, places, or activities. This information can also be recalled bilingually from sources in either of the languages handled by the speech translator.
The information extraction module (module 12 a) extracts key words and phrases from system logs (items 14 e and 14 f) generated by the speech recognition (modules 3, 8) and machine translation (modules 3, 8) components. First, usage statistics from recent system logs (step 123) are generated. Next, relevant information is obtained from the internet or local knowledgebase (item 125) based on keyword occurrence and language usage (step 124). This information is subsequently presented to the user (step 126) via the screen of the device (FIG. 1, item 13). Presented information will include targeted advertising and supportive information. Supportive information includes flight schedules, hotel availability and rates, and contact details of persons. Information can be presented bilingually in either language (La or Lb) by searching based on keywords in the output of machine translation (modules 3, 8).
receiving from a user, an utterance in a first language that is to be translated by a speech translation system from the first language to a second language;
receiving, an indication to add a new word in the first language to a first recognition lexicon of the first language of a first automatic speech recognition module of the speech translation system;
determining for the new word, by a processor, word class information, a pronunciation in the first language, a translation in the second language, and a pronunciation in the second language in response to receiving the indication to add the new word;
adding the new word the determined word class information and the determined pronunciation in the first language to the first recognition lexicon of the first language of the first automatic speech recognition module; and
adding the new word, the determined word class information, the determined translation in the second language and the pronunciation of the translation in the second language, to a first machine translation module associated with the first language of the speech translation system.
2. The method of claim 1, wherein determining word class information comprises estimating the word class information via a tagging model.
3. The method of claim 1, wherein determining a pronunciation in the first language comprises generating a pronunciation in the first language via either a rule-based model or a statistical model.
4. The method of claim 1, wherein determining a translation in the second language comprises generating a translation in the second language via either a rule-based model or a statistical model.
5. The method of claim 1, wherein the user is prompted to verify the word class information, pronunciation in the first language, and translation in the second language before the new word is added to the first recognition lexicon of the first language of the automatic speech recognition module.
6. The method of claim 1, wherein the user is prompted to verify the word class information, pronunciation in the first language, and translation in the second language before the new word is added to the first machine translation module associated with the first language of the speech translation system.
displaying simultaneously in text, on a user interface display of the speech translation system, at least (i) recognized speech in the utterance in the first language, and (ii) the translation into the second language of the speech in the utterance;
storing, by the speech translation system, a bilingual sentence-pair selected by the user via the user interface display, as a speech translation favorite, wherein the bilingual sentence pair comprises a sentence in the first language uttered by the user in a first dialog and a translation of the sentence from the first language into the second language; and
playing the translation of the sentence into the second language upon selection by the user to play the translation of the sentence into the second language in a second dialog that is after the first dialog, without the user having to speak the sentence in the second dialog.
8. A device-comprising:
at least one microphone for receiving an utterance in a first language from a user; and
a speech translation system in communication with the at least one microphone, wherein the speech translation system is for translating the utterance into a second language, and wherein the speech translation system comprises:
a first automatic speech recognition module for the first language for recognizing speech in the utterance in the first language;
a first machine translation module in communication with the first speech recognition module, wherein the first machine translation module is for translating the recognized speech in the first language, recognized by the first speech recognition module, into the second language; and;
a user interface in communication with the speech translation system for outputting the translation of the utterance in the second language determined by the first machine translation module;
wherein the speech translation system is configured to:
receive from the user, the utterance in the first language that is to be translated by the speech translation system from the first language to the second language;
receive an indication to add a new word in the first language to the first recognition lexicon of the first automatic speech recognition module of the speech translation system;
determine for the new word, by a processor, word class information, a pronunciation in the first language, the translation in the second language, and a pronunciation in the second language;
add the new word, the determined word class information and the determined pronunciation in the first language to the first recognition lexicon of the first language of the first automatic speech recognition module; and
add the new word, the determined word class information, the determined translation in the second language and the pronunciation of the translation in the second language, to the first machine translation module.
9. The device of claim 8, wherein determining word class information Comprises estimating the word class information via a tagging model.
10. The device of claim 8, wherein determining a pronunciation in the first language comprises generating a pronunciation in the first language via either a rule-based model Or a statistical model.
11. The device of claim 8, wherein determining a translation in the second Language comprises generating a translation in the second language via either a rule-based model Or a statistical model.
12. The device of claim 8, wherein the user is prompted to verify the word class information, pronunciation in the first language, and translation in the second language before the new word is added to the first recognition lexicon of the first language of the automatic speech recognition module.
13. The device of claim 8, wherein the user is prompted to verify the word class information, pronunciation in the first language, and translation in the second language before the new word is added to the first machine translation module associated with the first language of the speech translation system.
14. The device of claim 8, further comprising a text processor configured to identify words in the utterance as being potentially inappropriate and replacing the inappropriate words with a sound.
15. The device of claim 8, further comprising a conditioning mode configured to use the first translation module and first language module in combination with prosodic parameters to render the pronunciation of the translation more appropriate based on language use.
16. The device of claim 8, wherein the speech translation system:
a second automatic speech recognition module for recognizing speech in the second language; and
a second machine translation module for translating recognized speech in the second language to the first language: and
is further configured to identify whether the utterance is in the first language or the second language.
17. The device of claim 16, wherein the speech translation system is for:
automatically using the first automatic speech recognition module when it is identified by the speech translation system that an input utterance is in the first language; and
automatically using the second automatic speech recognition module when it is identified by the speech translation system that the input utterance is in the second language, without having the user input to the device who is speaking in a dialog between two speakers.
18. The device of claim 16, wherein the speech translation system further comprises an information extraction module for:
generating usage statistics from system logs generated by the first and second automatic speech recognition modules, and the first and second machine translation modules;
retrieving relevant information based on keyword occurrences in the usage statistics; and presenting the relevant information to the user via the user interface display.
19. The device of claim 8, wherein the device comprises a portable device.
20. The device of claim 8, wherein the user interface comprises a speaker.
21. The device of claim 20, wherein the user interface further comprises a display for displaying text of the translated utterance in the second language.
22. The device of claim 8, further comprising speech translation favorites module for:
storing a bilingual sentence-pair selected by the use via the user interface as a speech translation favorite of the user, wherein the bilingual sentence pair comprises a sentence in the first language uttered by the user in a first dialog and a translation of the sentence in the first language into the second language; and
23. The device of claim 22, wherein the user selects to play the translation of the sentence into the second language in the second dialog by clicking on an identifier for the bilingual sentence-pair via the user interface.
24. The device of claim 8, wherein the speech translation system further comprises a language learning module that is for:
building a profile of language usage statistics for the user based on usage by the user of the speech translation user device over a period of time, wherein the usage comprises utterances by the user in the first language; and
in a language learning mode, constructs a vocabulary drill for the user in the second language based on the profile of language usage statistics for the user.
25. The device of claim 24, wherein the language learning module is further for, in the language learning mode, generate a syntax drill for the user in the second language based on the profile of language usage statistics for the user.
US13008346 2007-10-26 2011-01-18 Enhanced speech-to-speech translation system and methods for adding a new word Active 2028-08-23 US8972268B2 (en)
US4507908 true 2008-04-15 2008-04-15
US9258108 true 2008-08-28 2008-08-28
US9389808 true 2008-09-03 2008-09-03
US12424311 US8204739B2 (en) 2008-04-15 2009-04-15 System and methods for maintaining speech-to-speech translation in the field
US12689042 US9070363B2 (en) 2007-10-26 2010-01-18 Speech translation with back-channeling cues
US13008346 US8972268B2 (en) 2008-04-15 2011-01-18 Enhanced speech-to-speech translation system and methods for adding a new word
US14522164 US20150046147A1 (en) 2008-04-15 2014-10-23 Translation system information extraction
US14589540 US9753918B2 (en) 2008-04-15 2015-01-05 Lexicon development via shared translation database
US15694733 US20180011842A1 (en) 2006-10-26 2017-09-01 Lexicon development via shared translation database
US12689042 Continuation US9070363B2 (en) 2006-10-26 2010-01-18 Speech translation with back-channeling cues
US12689042 Continuation-In-Part US9070363B2 (en) 2006-10-26 2010-01-18 Speech translation with back-channeling cues
US14522164 Continuation US20150046147A1 (en) 2007-10-26 2014-10-23 Translation system information extraction
US14589540 Continuation US9753918B2 (en) 2007-10-26 2015-01-05 Lexicon development via shared translation database
US20110307241A1 true US20110307241A1 (en) 2011-12-15
US8972268B2 true US8972268B2 (en) 2015-03-03
ID=53007655
US13008346 Active 2028-08-23 US8972268B2 (en) 2007-10-26 2011-01-18 Enhanced speech-to-speech translation system and methods for adding a new word
US14522164 Pending US20150046147A1 (en) 2007-10-26 2014-10-23 Translation system information extraction
US14589540 Active US9753918B2 (en) 2007-10-26 2015-01-05 Lexicon development via shared translation database
US15694733 Pending US20180011842A1 (en) 2006-10-26 2017-09-01 Lexicon development via shared translation database
US (4) US8972268B2 (en)
WO2014144800A1 (en) * 2013-03-15 2014-09-18 Cornell University Computer system methods for generating combined language content
JPH08329088A (en) 1995-05-31 1996-12-13 Sony Corp Speech input translation device
JP2002091963A (en) 2000-09-14 2002-03-29 Oki Electric Ind Co Ltd Machine translation system
US6697777B1 (en) 2000-06-28 2004-02-24 Microsoft Corporation Speech recognition user interface
US20050119884A1 (en) 2002-04-12 2005-06-02 Koninklijke Philips Electroncics N.V. Method and system for speech recognition of symbol sequences
US20080105769A1 (en) 2006-11-06 2008-05-08 Michael Ford Tschantz Compact check valve for spray nozzle
US20100217582A1 (en) 2007-10-26 2010-08-26 Mobile Technologies Llc System and methods for maintaining speech-to-speech translation in the field
JP2886121B2 (en) 1995-11-10 1999-04-26 株式会社エイ・ティ・アール音声翻訳通信研究所 Statistical language model generating apparatus and speech recognition apparatus
JP3875357B2 (en) 1996-08-02 2007-01-31 富士通株式会社 Word-phrase classification processing method, collocation extraction method, a word-phrase classification processing apparatus, the speech recognition apparatus, the machine translation apparatus, collocation extraction device and word-phrase storage medium
JP3441400B2 (en) 1998-06-04 2003-09-02 松下電器産業株式会社 Language conversion-rule making apparatus, and a program recording medium
JP4135316B2 (en) 2000-12-15 2008-08-20 ブラザー工業株式会社 Machine translation apparatus
JP2002207495A (en) 2001-01-11 2002-07-26 Nippon Hoso Kyokai <Nhk> Remote word additional registration system and method
JP3974412B2 (en) 2001-01-24 2007-09-12 松下電器産業株式会社 Speech conversion system
JP2004310256A (en) 2003-04-03 2004-11-04 Ricoh Co Ltd Translation dictionary management method and system, translation system, its program, and medium storing the program
JP4517698B2 (en) 2003-09-26 2010-08-04 三菱電機株式会社 The wavelength conversion laser device
WO2005050399A3 (en) 2003-11-14 2007-10-18 Speechgear Inc Phrase constructor for translator
JP4431759B2 (en) 2005-01-18 2010-03-17 株式会社国際電気通信基礎技術研究所 Unregistered word automatic extraction device, and a program, and unregistered word automatic registration apparatus and program
JP4702936B2 (en) 2005-06-28 2011-06-15 キヤノン株式会社 The information processing apparatus, control method, program
US8571849B2 (en) 2008-09-30 2013-10-29 At&T Intellectual Property I, L.P. System and method for enriching spoken language translation with prosodic information
US20100280828A1 (en) 2009-04-30 2010-11-04 Gene Fein Communication Device Language Filter
Aguero, P. et al., "Prosody Generation for Speech-to-=Speech Translation," International Conference on Acoust. Speech Signal Process, 2006, pp. I-566-I-560.
Arbabi et al., "Algorithms for Arabic Name Transliteration," IBM J. Res. Develop., Mar. 1994, vol. 38, No. 2.
Asadi, Ayman, Richard Schwartz, and John Makhoul. "Automatic modeling for adding new words to a large-vocabulary continuous speech recognition system." Acoustics, Speech, and Signal Processing, 1991. ICASSP-91., 1991 International Conference on. IEEE, 1991. *
Bach et al., "The Cmu Transtac 2007 Eyes-Free and Hands-Free Two-Way Speech-To-Speech Translation System," Interact, Language Technologies Institute, Carnegie Mellon University, 2007.
Brown et al., "The Mathmatics of Statistical Machine Translation: Parameter Estimation," Association for Computational Linguistics, 1993.
Chiang et al., "The Hiero Machine Translation System: Extensions, Evaluation, and Analysis," Proc. of Human Language Technology Conference on Empirical Methods in Natural Language Processing, Oct. 2005, pp. 779-786.
Collins, M. "Parameter Estimation for Statistical Parsing Models: Theory and Practice of Distribution-Free Methods," New Developments in Parsing Technology, Chapter 2, pp. 19-55, 2004, H. Bunt et al. (eds.), Kluwer Academic Publishers, Dordrecht, The Netherlands.
European Patent Office, Examination Report, European Patent Application No. 09732921.3, Sep. 9, 2013, five pages.
Fugen et al, "Open Domain Speech Translation: From Seminars and Speeches to Lectures," TC-Star Workshop on Speech-to-Speech Translation, Jun. 19-21, 2006, Barcelona Spain, pp. 81-86.
Fugen et al., "Advances in Lecture Recognition: The ISL RT-06S Evaluation System," INTERSPEECH 2006.
Fugen et al., "LingWear: A Mobile Tourist Information System," Proc. of Human Language Technology Conference, San Diego, CA 2001.
Fugen et al., "Open Domain Speech Translation: From Seminars and Speeches to Lectures," ICASSP, Toulouse, France 2006.
Gales, "Maximum Likelihood Multiple Projection Schemes for Hidden Markov Models," 2000 (abstract).
Gales, "Semi-Tied Covariance Matrices for Hidden Markov Models," ICASSP 1998 (abstract).
Gollan et al., "Cross Domain Automatic Transcription on the TC-STAR EPPS Corpus," ICASSP 2005.
H. Okuma et al., "Introducing Translation Dictionary into Phase-based SMT", Proc. Machine Translation Summit XI, (Sep. 10-14, 2007), pp. 361-368.
Hsieh, M. et al., "Combining lexical, syntactic and prosodic cues for improved online dialog act tagging," Computer Speech and Language, 2009, pp. 407-422, vol. 23.
Hurst, "Indexing, Searching, and Skimming of Multimedia Documents Containing Recorded Lectures and Live Presentations," 2003, In Proc. of ACM Multimedia, pp. 1-2.
International Preliminary Report on Patentability, PCT/US2011/021530, mailed Aug. 2, 2012.
Janin et al., "The ICSI Meeting Project: Resources and Research," Proc. of the ICASSP Meeting Recognition Workshop, Montreal, Canada, 2004.
Japan Patent Office, Office Action, Japanese Patent Application No. 2011-505169, Sep. 9, 2014, fourteen pages.
Kashioka, "Translation unit concerning timing of simultaneous translation" LREC-2002: Third International Conference on Language Resources and Evaluation, May 27-Jun. 2, 2002, Las Palmas de Gran Canaria, Spain, pp. 142-146.
Knight et al., "Machine Transliteration," Association for Computational Linguistics, 1998.
Koehm, "Europarl: A Multilingual Corpus for Evaluation of Machine Translation," Info. Sci. Inst. U. of S. CA, 2002.
Koehn et al., "Moses: Open Source Toolkit for Statistical Machine Translation," Proc. of the ACL 2007 Demo and Poster Sessions, Jun. 2007, pp. 177-180, Prague.
Koehn et al., "Open Source Toolkit for Statistical Machine Translation: Factored Translation Models and Confusion Network Decoding," Sep. 2007, one hundred two pages.
Köhler et al, "Rapid porting of ASR-systems to mobile devices," Proc. of the 9th European Conference on Speech Communication and Technology, Sep. 2005, pp. 233-236.
Köhler et al, "Rapid porting of ASR-systems to mobile devices," Proc. of the 9th European Conference on Speech Communication and Technology, Sept 2005, pp. 233-236.
Korean Intellectual Property Office, Office Action, Korean Patent Application No. 10-2010-7025487, Apr. 26, 2014, eight pages.
Lafferty et al., "Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data," http://www.cis.upenn.edu-pereira/papers/crf.pdf, 2001.
Lamel et al., "The Translanguage English Database (TED)," ICSLP, vol. LDC2002SO4, 1994 (abstract).
Lane, Ian R. et al., "Class-based Statistical Machine Translation for Field Maintainable Speech-to-Speech Translation," Interspeech, Sep. 22, 2008, pp. 2362-2365, http://isl.ira.uka.de/fileadmin/publication-files/851-IS081131.pdf.
Lane, Ian R. et al., "Class-based Statistical Machine Translation for Field Maintainable Speech-to-Speech Translation," Interspeech, Sep. 22, 2008, pp. 2362-2365, http://isl.ira.uka.de/fileadmin/publication-files/851—IS081131.pdf.
Lane, Ian R. et al., "Class-based Statistical Machine Translation for Field Maintainable Speech-to-Speech Translation," INTERSPEECH, Sep. 22, 2008, pp. 2362-2365, http://isl.ira.uka.de/fileadmin/publication-files/851-ISO81131.pdf.
Lane, Ian R. et al., "Class-based Statistical Machine Translation for Field Maintainable Speech-to-Speech Translation," INTERSPEECH, Sep. 22, 2008, pp. 2362-2365, http://isl.ira.uka.de/fileadmin/publication-files/851—ISO81131.pdf.
Leggetter et al., "Maximum Likelihood Linear Regression for Speaker Adaptation of Continuous Density Hidden Markov Models," Computer Speech & Lang., vol. 9(2), 1995 (abstract).
Litman, D. et al., "Predicting Automatic Speech Recognition Performance Using Prosodic Cues," Proceedings of NAACL, 2000, pp. 218-225.
Mangu et al., "Finding Consensus Among Words: Lattice-Based Word Error Minimization," EUROSPEECH, 1999.
Matusov et al, "Evaluating Machine Translation Output with Automatic Sentence Segmentation," Proc. of IWSLT 2005, pp. 148-154, Oct. 2005.
Metze et al. "The Nespole! Speech-To-Speech Translation System," Proceeding of HLT 2002, Second International Conference on Human Language Technology Research, San Francisco, 2002.
Nanjo et al, "Language Model and Speaking Rate Adaptation for Spontaneous Presentation Speech Recognition," IEEE Transactions of Speech and Audio Processing, Jul. 2004, vol. 12, No. 4, pp. 391-400.
Narayanan, S. et al., "Transonics: A Speech to Speech System for English-Persian Interactions," Automatic Speech Recognition and Understanding, 2003. ASRU 2003, 2003 IEEE Workshop on St. Thomas, VI, USA, Nov. 30-Dec. 3, 2003. Piscataway, NJ, USA, IEEE, Nov. 30, 2003, pp. 670-675.
Niemann, H. et al., "Using Prosodic Cues in Spoken Dialog Systems," Proceedings of SPECOM '98 Workshop, 1998, pp. 17-28.
Och et al., "A Systematic Comparison of Various Statistical Alignment Methods," Association for Computational Linguistics, 2003.
Och et al., "Improved Alignment Models for Statistical Machine Translation," http://www.aclweb.org/anthology-new/W/W99/W99-0604.pdf, 1999.
Olszewski, et al., "Highly Directional Multi-Beam Audio Loudspeaker," INTERSPEECH 2006 (abstract).
Papineni et al., "Bleu: A Method for Automatic Evaluation of Machine Translation," IBM Research Report, RC22176 (W0109-022), Sep. 17, 2001.
Papineni et al., "Bleu: A Method for Automatic Evaluation of Machine Translation," IBM Research Report, RC22176 (W0I09-022), Sep. 17, 2001.
Papineni et al., "Bleu: A Method for Automatic Evaluation of Machine Translation," Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Jul. 2002, pp. 311-318.
Paulik et al, "Document driven machine translation enhanced ASR," European Conference on Speech Communication and Technology, Interspeech, Sep. 2005, pp. 1-4.
PCT International Search Report and Written Opinion, PCT Application No. PCT/US2009/040677, Jul. 20, 2009, eight pages.
PCT International Search Report and Written Opinion, PCT Application No. PCT/US2011/021530, Mar. 28, 2011, ten pages.
Rao et al., "Improving Spoken Language Translation by Automatic Disfluency Removal: Evidence from Conversational Speech Transcripts," Machine Translation Summit XI, Copenhagen, Denmark, Oct. 9, 2007, 5 pages.
Soltau et al., "A One-Pass Decoder Based on Polymorphic Linguistic Context Assignment," Automatic Speech Recognition and Understanding, 2001 (abstract).
Stolcke, "SRILM-An Extensible Language Modeling Toolkit," Speech Technology and Research Laboratory, SRI International, Menlo Park, CA, 2002.
Stolcke, "SRILM—An Extensible Language Modeling Toolkit," Speech Technology and Research Laboratory, SRI International, Menlo Park, CA, 2002.
Stolcke, A., "SRILM-An Extensible Language Modeling Toolkit," http://www.speech.sri.com/cgi-bin/run-distill?papers/icslp2002-srilm.ps.gz, 2002.
Stolcke, A., "SRILM—An Extensible Language Modeling Toolkit," http://www.speech.sri.com/cgi-bin/run-distill?papers/icslp2002-srilm.ps.gz, 2002.
Stuker et al., "The ISL TC-STAR Spring 2006 ASR Evaluation Systems," TC-STAR Speech-to-Speech Translation Workshop, Spain, 2006.
Suhm et al., "Towards Beder Language Models for Spontaneous Speech," 1994, Carnegie Mellon University, four pages.
TC-STAR, "Technologies and Corpora for Speech-to-Speech Translation," 2004.
United States Office Action, U.S. Appl. No. 12/689,042, Jun. 19, 2013, nine pages.
United States Office Action, U.S. Appl. No. 12/689,042, Jun. 20, 2014, fourteen pages.
United States Office Action, U.S. Appl. No. 12/689,042, Oct. 28, 2014, fifteen pages.
United States Office Action, U.S. Appl. No. 13/008,346, Sep. 18, 2013, twenty-one pages.
Vogel et al, "The ISL statistical translation system for spoken language translation," Proc. of the International Workshop on Spoken Language Translation, Kyoto, Japan, 2004, pp. 65-72.
Waibel et al., "Class-Based Statistical Machine Translation for Field Maintainable Speech-to-Speech Translation," InterSpeech 2008, Brisbane, Australia, Sep. 22, 2008, 4 pages.
Waibel et al., "Computers in the Human Interaction Loop," 5th International Workshop on Image Analysis for Multimedia Interactive Services, Lisbon 2004.
www.ldc.upenn.edu, Linguistic Data Consortium, Nov. 5, 2007.
www.ssli.ee.washington.edu/projects/ears/WebData/web-data-collection.html, Web Data for Language Modeling, Nov. 5, 2007.
www.ssli.ee.washington.edu/projects/ears/WebData/web—data—collection.html, Web Data for Language Modeling, Nov. 5, 2007.
Yamada et al., "A Decoder for Syntax-Based Statistical Mt," Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Jul. 2002, pp. 303-310.
Yarwsky et al., "Inducing Multilingual Text Analysis Tools Via Robust Projections Across Algined Corpora," http://delivery.acm.org.10.1145.1080000/1072187/p35-yarowsky.pdf., 2001.
Zhang, Y. et al., "PanDoRA: A Large-scale Two-way Statistical Machine Translation System for Hand-held Devices," Proceedings of MT Summit XI, Sep. 10-14, 2007, Copenhagen, Denmark, eight pages.
Zhao et al., "A Log-Linear Block Transliteration Model Based on Bi-Stream Hmms," Proceedings of NAACL HLT, 2007, pp. 364-371.
US9753918B2 (en) 2017-09-05 grant
US20110307241A1 (en) 2011-12-15 application
US20150127321A1 (en) 2015-05-07 application
US20150046147A1 (en) 2015-02-12 application
US20180011842A1 (en) 2018-01-11 application