Source: https://patents.google.com/patent/US9753918B2/en
Timestamp: 2019-06-27 10:52:38
Document Index: 486901977

Matched Legal Cases: ['Application No. 14190061', 'Application No. 11702324', 'Application No. 14190061', 'Application No. 14190061', 'Application No. 09732921', 'Application No. 2011', 'Application No. 2015']

US9753918B2 - Lexicon development via shared translation database - Google Patents
Lexicon development via shared translation database Download PDF
US9753918B2
US9753918B2 US14/589,540 US201514589540A US9753918B2 US 9753918 B2 US9753918 B2 US 9753918B2 US 201514589540 A US201514589540 A US 201514589540A US 9753918 B2 US9753918 B2 US 9753918B2
US14/589,540
US20150127321A1 (en
2011-01-18 Priority to US13/008,346 priority patent/US8972268B2/en
2015-01-05 Priority to US14/589,540 priority patent/US9753918B2/en
2015-01-05 Application filed by Facebook Inc filed Critical Facebook Inc
2015-05-07 Publication of US20150127321A1 publication Critical patent/US20150127321A1/en
2017-09-05 Publication of US9753918B2 publication Critical patent/US9753918B2/en
In an embodiment, a language learning module, 12 b, will be present which extracts information from recent system logs (14 e and 14 l) to subsequently be used for language learning. Drills are automatically generated based on recent system usage by the user, additionally the user can choose any sentence output from the system to add to his/her active learning drill (step 34 c).
Similarly, to add a new Japanese word “Wakayama” (item 70) to the system, first the Japanese pronunciation is generated via manually defined rules (step 71). Next, the transliteration of this word in Japanese is automatically generated via rule-based transliteration (step 72), the English pronunciation is then generated via manually defined rules (step 73). The rule based transliteration may be performed using methods of Mansur Arbabi, Scott M. Fischthal, Vincent C. Cheng, and Elizabeth Bar, “Algorithms for Arabic name transliteration,” IBM Journal of research and Development, 38(2):183
193, 1994. The resulting information (item 74) is then verified by the user before registering the word into the active system vocabulary.
One embodiment of class-based machine translation is class-based statistical machine translation, in which a foreign language sentence f1 J=f1, f2, . . . , fJ is translated into another language eI 1=e1, e2, . . . , eI by searching for the hypothesis ^eI 1 with maximum likelihood, given:
^e I 1=argmax P(e I 1 |f J 1)=argmax P(f J 1 |e I 1)=·P(e I 1)
@PLACE.city {
□□□@TIME(4:30}
Through experimental evaluation, we show that class-based machine translation, as detailed above, improves translation performance compared to previous approaches. Furthermore, we show that by using the parallel tagging approach described above, translation accuracy is further improved.
In addition to the basic speech translation system shown in FIG. 1 an embodiment of this system will add additional functionality to allow users to playback predefined phrases via TTS (FIG. 1, modules 3 and 7). An example graphical user interface for this functionality is shown in FIG. 16. While using the speech translation system the users can select any output generated by the system and add this to their favor list (step 34 b). The system will check if the sentence-pair already exists in the “Translation Favorites” list (step 109) and if not will append it (step 109 a). A possible graphical user interface for this phrase-addition step is shown in FIG. 13. In this example the user can press a graphical button (item 127) to add the current bilingual sentence-pair to the user's “Translation Favorites” list. Instead of performing speech translation, the user can enter the “Translations Favorites” mode (step 15 f). In this mode (detailed in FIG. 13) The system allows users to select a sentence-pair for playback (steps 110, 111), edit sentence-pairs currently in the list (steps 112,113), delete a sentence-pair from the list (steps 114, 115) or append a sentence-pair to the list (steps 116, 117) by typing in the required information.
The speed of the speech translation system shown in FIG. 1 is dependent on the vocabulary size of the speech recognition models (item 17) applied. In an embodiment of the system recognition is performed with a medium sized recognition vocabularies consisting of around 40,000 entries. This vocabulary size provides reasonable coverage over possible words uttered by users and allows speech recognition (step 27) to be performed reasonably fast. In addition to this method, an additional embodiment could be realized using a much smaller initial recognition vocabulary (item 20) and language model (item 19) consisting of less than 10,000 vocabulary entries. When the user identifies a new word using the Correction and Repair Module (module 11) the appropriate speech recognition models (item 17) will be updated via the User Field Customization Module (module 12). This procedure is shown in FIG. 3, steps 35, 36 and 37. Once identified the new word is incorporated into the system as shown in FIG. 5.
receiving from a user an utterance in a first language including a first term;
by a speech translation system, translating the utterance from the first language into a second language;
receiving an indication from the user to initiate a user customization process for customizing one or more modules of the speech translation system to the user; and
under the user customization process:
receiving an indication from the user to add the first term to one or more modules of the speech translation system;
in response to the received indication from the user, determining word class information for the first term;
adding, by the speech translation system, the first term, the determined word class information, and at least a portion of the translation of the utterance in the second language to a first machine translation module associated with the first language of the speech translation system; and
adding the first term and the at least a portion of the translation of the utterance in the second language to a shared database for a community such that the customization performed by the user is available for use by other users of the community in translations by the speech translation system.
retrieving a second term from the shared database; and
adding the second term from the shared database to the first machine translation module.
3. The method of claim 1, wherein the at least a portion of the translation of the utterance in the second language added to the shared database comprises an incorrect or incomplete translation, and further comprising:
requesting, with the first term added to the shared database, a translation correction for the first term.
retrieving a corrected translation of the first term from the shared database; and
adding the corrected translation from the shared database to the first machine translation module.
5. The method of claim 4, wherein the corrected translation of the first term in the shared database is provided by a second user of the community.
displaying simultaneously in text, on a user interface display of the speech translation system, at least (i) the received utterance in the first language, and (ii) the translation of the utterance in the second language.
8. The method of claim 1, wherein translating the utterance in the first language into the second language comprises generating a translation in the second language via either a rule-based model or a statistical model.
in response to the received indication from the user, determining a pronunciation in the first language for the first term; and
adding the first term, the determined word class information, and the pronunciation in the first language to a first recognition lexicon of the first language of a first automatic speech recognition module of the speech translation system.
saving the utterance and the translation as sentence pairs upon instruction by the user to save the sentence pairs as a favorite in a speech translation favorites module configured to store a list or hierarchical inventory of such sentence pairs wherein each favorite can be customized and played directly upon user selection in either the first or second language.
adding the pronunciation in the first language to the shared database.
in response to the received indication from the user, determining a pronunciation in the second language for the first term; and
adding the pronunciation in the second language to the shared database.
translating, by a speech translation system, the utterance in the first language into a second language;
adding by the speech translation system the first term, the determined word class information, and at least a portion of the translation of the utterance in the second language to a first machine translation module associated with the first language of the speech translation system; and
14. The computer program product of claim 13, further comprising computer program code for performing the steps:
15. The computer program product of claim 13, wherein the translation in the second language added to the shared database comprises a poor translation, further comprising computer program code for performing the step:
16. The computer program product of claim 13, further comprising computer program code for performing the steps:
adding the corrected translation from the shared database to the first machine translation module as.
17. The computer program product of claim 13, wherein the corrected translation of the first term in the shared database is provided by a second user of the community.
18. The computer program product of claim 17, wherein the shared database stores terms associated with one or more fields comprising one or more of: names, expressions, special terms.
19. The computer program product of claim 13, further comprising computer program code for performing the step:
20. The computer program product of claim 13, wherein translating the utterance in the first language into the second language comprises generating a translation in the second language via either a rule-based model or a statistical model.
21. The computer program product of claim 13, further comprising computer program code for performing the steps:
an automatic speech recognition module for receiving from a user, an utterance in a first language including a first term associated with a field term;
a translation module for translating the utterance from the first language into a second language;
a user field customization module for receiving an indication from the user to initiate a user customization process for customizing one or more modules of the speech translation system to the user; and under the user customization process:
for receiving an indication from the user to add the first term to one or more modules of the speech translation system,
for, in response to the received indication from the user, determining word class information the first term,
for adding by the speech translation system the first term, the determined word class information, and at least a portion of the translation of the utterance in the second language to a first machine translation module associated with the first language of the speech translation system; and
for adding the first term and the at least a portion of the translation of the utterance in the second language to a shared database for a community such that the customization performed by the user is available for use by other users of the community in translations by the speech translation system.
US14/589,540 2007-10-26 2015-01-05 Lexicon development via shared translation database Active US9753918B2 (en)
US15/694,733 Continuation US20180011842A1 (en) 2006-10-26 2017-09-01 Lexicon development via shared translation database
US20150127321A1 US20150127321A1 (en) 2015-05-07
US9753918B2 true US9753918B2 (en) 2017-09-05
US13/008,346 Active 2028-08-23 US8972268B2 (en) 2007-10-26 2011-01-18 Enhanced speech-to-speech translation system and methods for adding a new word
US14/522,164 Pending US20150046147A1 (en) 2007-10-26 2014-10-23 Translation system information extraction
US14/589,540 Active US9753918B2 (en) 2007-10-26 2015-01-05 Lexicon development via shared translation database
US15/694,733 Pending US20180011842A1 (en) 2006-10-26 2017-09-01 Lexicon development via shared translation database
US20180107655A1 (en) * 2016-10-17 2018-04-19 Google Inc. Systems and methods for handling formality in translations of text
2011-01-18 US US13/008,346 patent/US8972268B2/en active Active
2014-10-23 US US14/522,164 patent/US20150046147A1/en active Pending
2015-01-05 US US14/589,540 patent/US9753918B2/en active Active
2017-09-01 US US15/694,733 patent/US20180011842A1/en active Pending
Benus, S. et al., "The Prosody of BackChannels in American English," ICPhS XVI, Aug. 6-10, 2007, pp. 1065-1068.
Bertrand, R. et al., "Backchannels revisited from a multimodal perspective," Proceedings of Auditory-visual Speech Processing, 2007, five pages, Hilvarenbeek, Netherlands.
Cathcart, N. et al., "A Shallow Model of Backchannel Continuers in Spoken Dialogue," Proceedings of the Tenth Conference on European Chapter of the Association for Computational Linguistics, 2003, pp. 51-58, vol. 1, Association for Computational Linguistics.
European Patent Office, European Search Report and Opinion, European Patent Application No. 14190061.3, Jul. 2, 2015, ten pages.
European Patent Office, Examination Report, European Patent Application No. 11702324.2, Sep. 28, 2015, four pages.
European Patent Office, Examination Report, European Patent Application No. 14190061.3, Aug. 12, 2016, seven pages.
European Patent Office, Partial European Search Report, European Patent Application No. 14190061.3, Mar. 2, 2105, six pages.
European Patent Office, Summons to attend oral proceedings pursuant to Rule 115(1) EPC, Aug. 4, 2016, seven pages.
European Patent Office, Summons to attend oral proceedings pursuant to Rule 115(1) EPC, European Patent Application No. 09732921.3, Mar. 17, 2015, five pages.
Fugen, C. et al., "Open Domain Speech Translation: From Seminars and Speeches to Lectures," TC-STAR Workshop on Speech-to-Speech Translation, Jun. 19-21, 2006, pp. 81-86, Barcelona, Spain.
Fujie, S., et al., "A Conversation Robot with Back-channel Feedback Function based on Linguistic and Nonlinguistic Information," Second International Conference on Autonomous Robots and Agents, Dec. 13-15, 2004, pp. 379-384, Palmerston North, New Zealand.
Hamaker, J. et al., "Resegmentation and Transcription of Switchboard," in Proceedings of LVCSR Workshop, Sep. 1998, four pages, Maritime Institute of Technology, Linthicum Heights, Maryland, United States.
Hurst, W., "Indexing, Searching, and Skimming of Multimedia Documents Containing Recorded Lectures and Live Presentations," Proceedings of ACM Multimedia, Nov. 2-8, 2003, two pages, Berkeley, USA.
I R LANE , WAIBEL A: "Class-based Statistical Machine Translation for Field Maintainable Speech-To-Speech Translation", INTERSPEECH 2008, 22 September 2008 (2008-09-22) - 26 September 2008 (2008-09-26), pages 2362 - 2365, XP002535596
Japan Patent Office, Office Action, Japanese Patent Application No. 2011-505169, Jul. 7, 2015, twenty-three pages.
Japan Patent Office, Office Action, Japanese Patent Application No. 2015-218066, Feb. 28, 2017, eight pages.
Kashioka, H., "Translation Unit Concerning Timing of Simultaneous Translation," Proceedings of the Third International Conference on Language Resources and Evaluation, May 27-Jun. 2, 2002, pp. 142-146, Las Palmas de Gran Canaria, Spain.
Kohler, T.W. et al., "Rapid Porting of ASR-Systems to Mobile Devices," Proceedings of the 9th European Conference on Speech Communication and Technology, Sep. 2005, pp. 233-236, Lisbon, Portugal.
Lane, I.R. et al., "Class-based Statistical Machine Translation for Field Maintainable Speech-To-Speech Translation," Interspeech 2008, Sep. 26, 2008, pp. 2362-2365, XP002535596.
Leggetter, C.J. et al., "Maximum likelihood linear regression for speaker adaptation of continuous density hidden Markov models," Proceedings of Computer Speech and Language, 1995, pp. 171-185, vol. 9.
Matusov, E. et al., "Evaluating Machine Translation Output with Automatic Sentence Segmentation," Proceedings of IWSLT, 2005, pp. 148-154, Pittsburgh, USA.
Nanjo, H. et al., "Language Model and Speaking Rate Adaptation for Spontaneous Presentation Speech Recognition," IEEE Transactions on Speech and Audio Processing, Jul. 2004, pp. 391-400, vol. 12.
Noguchi, H. et al., "Prosody-based detection of the context of backchannel responses," ICSLP, 1998, four pages.
Okumura, A. et al., "An Automatic Speech Translation System for Travel Conversation," Second International Conference on Human Language Technology Research, 2002, pp. 411-412, vol. 43, No. 1.
Paulik, M. et al., "Document Driven Machine Translation Enhanced ASR," Proceedings of European Conference on Speech Communication and Technology, Interspeech, Sep. 2005, four pages.
Schaaf, T., "Detection of OOV Words Using Generalized Word Models and a Semantic Class Language Model," in Proceedings of Eurospeech, 2001, four pages.
Shriberg, E. et al., "Prosody-Based Automatic Segmentation of Speech into Sentences and Topics," Speech Communication, 2000, pp. 127-154.
Stolcke, A., "Modeling Linguistic Segment and Turn Boundaries for N-Best Rescoring of Spontaneous Speech," In Proceedings of Eurospeech, 1997, vol. 5, four pages.
Stolke, A. et al., "Automatic Linguistic Segmentation of Conversational Speech," Fourth International Conference on Spoken Language, ICSLP '96, Oct. 3, 1996-Oct. 6, 1996, pp. 1005-1008, vol. 2, Philadelphia, U.S.A.
Stolke, A. et al., "Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech," Computational Linguistics, 2000, pp. 339-373, vol. 26, No. 3.
United States Office Action, U.S. Appl. No. 13/675,775, Nov. 21, 2014, sixteen pages.
United States Office Action, U.S. Appl. No. 14/522,164, Mar. 9, 2016, twelve pages.
United States Office Action, U.S. Appl. No. 14/722,013, Oct. 3, 2016, thirteen pages.
United States Office Action, U.S. Appl. No. 14/797,045, Sep. 18, 2015, twenty-nine pages.
Vogel, S. et al., "The ISL Statistical Translation System for Spoken Language Translation," Proceedings of the International Workshop on Spoken Language Translation, 2004, pp. 65-72, Kyoto, Japan.
Waibel, A. et al., "Spoken Language Translation," IEEE Signal Processing Magazine, May 2008, pp. 70-79, vol. 25, No. 3.
Ward, N. "Using Prosodic Clues to Decide When to Produce Back-channel Utterances," Proceedings of the Fourth International Conference on Spoken Language, ICSLP 96, Oct. 3-6, 1996, pp. 1728-1731, vol. 3, Philadelphia, PA, USA.
Ward, N. et al., "Prosodic features which cue back-channel responses in English and Japanese," Journal of Pragmatics, 2000, pp. 1177-1207, vol. 32.
US8972268B2 (en) 2015-03-03
US20110307241A1 (en) 2011-12-15
US20150127321A1 (en) 2015-05-07
US20150046147A1 (en) 2015-02-12
US20180011842A1 (en) 2018-01-11