Graham Neubig 1 , Sakriani Sakti 1 , Tomoki Toda 1 , Satoshi Nakamura 1 , Yuji Matsumoto¹, Ryosuke Isotani², Yukichi Ikeda² 1 Graduate School of Information Science, Nara Institute of Science and Technology 8916-5 Takayama-cho, Ikoma-shi, Nara, Japan {neubig,ssakti,tomoki,s-nakamura,matsu}@is.naist.jp ² NEC Corporation 5-7-1 Shiba, Minato-ku, Tokyo, Japan {r-isotani@bp,y-ikeda@df}.jp.nec.com
Abstract
In this paper, we describe the overall de- sign for a speech translation system that aims to reduce the problems caused by lan- guage barriers in medical situations.  As first steps to building a system according to this design, we describe a collection of a medical corpus, and some translation ex- periments performed on this corpus. As a result of the experiments, we find that the best of three modern translation systems is able to translate 33%-81% of the sentences in a way such that the main content is un- derstandable.
1
Introduction
One of the most important elements to provision of high-quality medical service is communication between medical practitioners and patients. How- ever, in situations where practitioners and patients do not share a common language, the language barrier prevents effective communication, making proper diagnosis and treatment much more diffi- cult. Language barriers occur in medical situations with immigrants who may speak the language of their country of residence to some extent, but not enough to effectively communicate medical symp- toms.  There is also the case of medical tourism, where tourists may visit another country to receive high-quality or affordable medical treatment that is not available in their home country. One potential method for overcoming the com- munication barrier in medical situations is through the use of automatic speech translation technol- ogy (Nakamura, 2009).  Automatic translation of speech in medical situations can be expected to be challenging for a number of reasons.  The first reason is that communication of incomplete or in- correct information could lead to a mistaken di- agnosis with severe consequences, and thus ex- tremely high levels of accuracy and reliability are required. The second reason is that conversation in the medical domain has its own unique vocab- ulary and expressions, and thus it is natural to as- sume that we must adapt the system appropriately to the medical domain. There has been some previous work attempting to adapt communication technology to meet these two challenges.  Eck et al. (2004) focus on adapt- ing a translation system to medical vocabulary, although the focus on text translation of medical documents instead of speech translation for com- munication.  Miyabe et al. (2007) propose a sys- tem for reliable multilingual communication, but rely on a graphical interface that is something like a powerful bilingual phrasebook adapted to com- munication at a hospital reception desk. In this paper, we describe our vision for full speech translation in medical situations, and some first steps to achieve this vision.  First, in Section 2 we describe our overall design for the speech translation system. This system includes the com- mon components of automatic speech recogni- tion (ASR), machine translation (MT), and text- to-speech (TTS), augmented to adapt each com- ponent to the task at hand. We also consider what is necessary to ensure the reliability of translation results, and consider the use of a system to allow the conversation to be forwarded to human medi- cal interpreters when necessary. In the first step towards achieving a transla- tion system for the medical domain we have also collected a medical-domain corpus for Japanese- English and Japanese-Chinese translation, as de- scribed in Section 3.  We share some insights gained in collecting this corpus, particularly com- paring and contrasting text data from a medical domain bilingual phrasebook, and actual conver- sational data gathered during doctors' visits. Based on this data, we then build several pro- totype translation systems for the four language pairs as described in Section 4.  We perform au- tomatic and manual evaluation of the results and evaluate how close we are to our goal of creat- ing a system that can provide a first wave of as- sistance in medical situations.  In particular, we find that over all four (relatively difficult) language pairs, we are on our way towards creating a practi- cal medical machine translation system, with from 33%-81% of sentences over two tasks and four language pairs having all content understandable with some effort. Finally, in Section 5 we conclude the paper with a discussion of future work.
Medical Translation System Design
2
We show the overall use scenario in Figure 1.  In a typical doctor's visit, patient first enters the doc- tor’s office, speaks with receptionists, and fills out forms. The patient will then enter the doctor's of- fice and communicate with doctors/nurses. In or- der to introduce a speech translation system for use in this scenario, we will provide a device that will then translate between the language of the pa- tient and that of the medical practitioners. The device provides two possible methods of communication.  The first is through the use of automatic speech translation technology, where the speaker's voice will be recorded, recognized, translated, and synthesized entirely automatically. In addition, as speech translation technology is still far from perfect, the device will also have the ability to connect to an actual human medical in- terpreter located in a call center.  However, as the cost of hiring and maintaining medical interpreters is quite high, we would also like to reduce our re- liance on human effort as much as possible. Thus, each device will use automatic translation by de- fault, but also have functionality to connect to hu- man interpreters, either based on a manual request of one of the users, or through automatic detection of when the dialogue is going poorly, such as the method described by Walker et al. (2000). Even with this fall-back to human interpreters, it is still desirable that the automatic translation system is effective as possible. In order to ensure this, we must be certain that the ASR, MT, and TTS models are all tuned to work as well as possi- ble in medical situations. Some potential problems that we have identified so far based on our analysis of data are as follows: Specialized Vocabulary:  Perhaps the most obvi- ous problem is that the ASR, MT, and TTS systems must all be able to handle the spe- cialized vocabulary and usage that occurs in the medical domain.  For example, most we found that unadapted systems had trouble handling specialized terms such as "cardio- gram," medicine names such as "Sudafed," and disease names such as "chicken pox." This will require the creation of domain spe- cific corpora/dictionaries, and domain adap- tation for each of the components (Leggetter and Woodland, 1995; Bellegarda, 2004). Conversational Speech:  The speech during doc- tor's visits will generally be somewhat in- formal and conversational when compared to that of speeches, news, or other more formal locations.  As a result, we can expect ASR to be more difficult due to fillers, disfluencies and other factors (Goldwater et al., 2010). Translation/Synthesis of Erroneous Input:  As we can expect ASR not to be perfect, it will be necessary to be able to translate input that contains errors. This problem can potentially be ameliorated by passing multiple speech recognition hypotheses to translation (Ney, 1999), and jointly optimizing the parameters of ASR and MT (Zhang et al., 2004; Ohgushi et al., 2013).  In addition, it will also be necessary to resolve difficulties in TTS due to grammatical errors, lack of punctuation, and unknown words (Parlikar et al., 2010). While all of these problems need to be solved to provide high-reliability speech translation sys- tems, in this paper as a first step we focus mainly on the MT system, and relegate the last problem of integration with ASR to future work.
3
Medical Translation Corpus
Construction and Analysis
In this section, we describe our collection of a tri-lingual (Japanese, English, Chinese) corpus to serve as an initial testbed for our medical transla- tion experiments, and an analysis of the corpus. 3.1
Corpus Construction
In general, when creating a corpus for train- ing/testing a machine translation system, it is im- portant to collect content that is as close as possi- ble to that which we will encounter when the sys- tem is actually used. In our medical translation sit- uation, this is true for both vocabulary (the corpus must cover special medical terms) and for speak- ing style (the corpus must have a similar style to that used by actual doctors and patients speaking through the system).  There is also the practical concern that the cost of corpus collection is high, so we would like to perform collection in efficient a manner as possible. Based on these principles, we designed and col- lected the following two corpora: Medical Phrasebooks: The first corpus consists of sentences designed based on sentences from Japanese-English bilingual phrase- books designed for interpreters focusing on the medical domains.  Chinese translations were obtained by translating each phrase from Japanese to Chinese.  This corpus has the advantages of relatively efficient con- struction, and good coverage of medical- domain terminology, but the conversations are not necessarily exactly representative of the conversations that actually occur at a doc- tor's office. Medical Conversation: The second corpus we gathered consists of actual conversations be- tween the patient and the receptionists or doc- tors recorded during a doctor's visit. The doctors and receptionists were all actual prac- ticioners, but for privacy reasons the person acting as a patient was actually healthy, but given a scenario to act out.  Conversations were recorded in Japanese and all partici- pants were native Japanese speakers.  The conversations were then segmented by utter- ance and translated into English and Chinese. This corpus has the advantage of being highly
natural and covering medical domain termi- nology, but requires a large amount of time and effort for the creation of scenarios, gath- ering the participants, execution of the actual dialog, and transcription/translation of the re- sults. At the end of the collection, we had 5130 and 1007 sentences for the phrasebook and conversa- tion corpus respectively.  In addition, we create three splits of the corpus for use in the training, tuning, testing of our machine translation system with a ratio of 4:1:1. The final size of the data in all of these corpora is shown in Table 1.
3.2
Corpus Analysis
In this section, we describe some insights gained from the analysis of both corpora, with some ex- amples illustrating the features of each corpus in Table 2. One feature of the data with major implica- tions is that there were large differences in speak- ing style between the phrasebook and conversa- tion corpora.  The data in the phrasebook corpus generally consisted of longer sentences, while the
Table 2: Examples of sentences (or several sentences separated by slashes) from the phrasebook and conversational corpora. majority of the utterances in the conversation cor- pus contained short questions, requests, responses, and commands.  This trend of longer sentences in the phrasebook corpus is shown clearly in Figure 2, which shows the cumulative length distribution of English sentences under a certain length in both corpora.  Focusing on sentences under length 15, we can see that this covers a total of 95% of the conversation corpus, but only 72% of the phrase- book corpus. In addition, the language in the conversation corpus is significantly less formal, particularly in Japanese where spoken language includes features such as dropped subjects or particles and abbre- viations, which rarely occur in written language (Neubig et al., 2012).  We hypothesize that in a cross-lingual medical conversation situation, the content of the utterances will fall somewhere be- tween these two situations, as the content will be conversational, but the kind of natural and infor- mal interaction seen two native speakers will be difficult to achieve through an automatic transla- tion system. The second enlightening feature of the two cor- pora that we noticed was that medical terminology was significantly less prevalent in the conversation corpus. This is also natural, as actual patients to a doctors office will likely be unfamiliar with diffi- cult medical terms, and thus the doctors will tend to explain in language that is understandable for their audience.  This observation will likely carry over to computer-mediated medical communica- tion as well.  As a result, it is likely that adapting to medical terminology of the domain is somewhat less important than adapting to the conversational speaking style of the speech.
Preliminary Evaluation of Medical
4
Machine Translation
In this section we describe a preliminary evalua- tion of the effectiveness of automatic translation on the medical domain data described in the pre- vious section.  In particular, we focus on the MT component, leaving evaluation of ASR, TTS, and the system as a whole for future work.
4.1
Experimental Setup
For the tuning and test data for our translation system, we use the data described in the previ- ous section.  For training, 4,000 sentences is not enough to build an accurate MT system, so we add several additional corpora for each language pair.  For Japanese-English parallel training data, we add the Eijiro dictionary 1 and its accompany- ing sample sentences, the BTEC corpus(Takezawa et al., 2002), and Wikipedia data from the Ky- oto Free Translation Task (Neubig, 2011), for a total of 1.33M parallel sentences and 1.97M dictionary entries.  For Japanese-Chinese paral- lel training data, we add a dictionary extracted from Wikipedia's language links 2 , the BTEC cor- pus, and TED talks (Cettolo et al., 2012) for a total of 519k sentences and 184k dictionary en- tries.  In addition, we add monolingual from En- glish GigaWord with 22.5M sentences and Chi- nese Wikipedia with 841k sentences. We compare three different statistical transla- tion methodologies: phrase-based MT (PBMT, (Koehn et al., 2003)), hierarchical phrase-based MT (Hiero, (Chiang, 2007)), and forest-to-string MT (F2S, (Mi et al., 2008)). The reason why we test these three methodologies is because the for- mer two methodologies do not rely on syntactic analysis, and thus may be more robust to con- versational input that is ill-formed and/or infor- mal.  On the other hand, using syntactic informa- tion has been shown to improve translation, partic- ularly between language pairs with different syn- tactic structures such as those we are handling in our experiments.  Thus it will be interesting to see which methodology can produce better results, and also if any difference in the effectiveness of the methodologies will be seen between the two corpora. For training the translation models and decod- ing, we use the Moses toolkit (Koehn et al., 2007) for PBMT and Hiero, and the Travatar (Neubig, 2013) toolkit for F2S with the default settings. For training language models, we use SRILM (Stol- cke, 2002), training Kneser-Ney smoothed 5-gram models for each individual language model train- ing corpus, and linearly interpolating these models to maximize likelihood on the tuning corpus. For tokenization we use the Stanford Tok- enizer/Segmenter for English and Chinese (Tseng et al., 2005), and the KyTea segmenter for Japanese (Neubig et al., 2011).  For syntactic parsing in English and Chinese we use a modi- fied version of the Egret parser, 3 and for Japanese we use the Eda parser (Flannery et al., 2011) and the dependency-to-CFG conversion rules in the Travatar toolkit.  Alignment is performed us- ing the unsupervised aligner GIZA++ (Och and Ney, 2003) for Japanese-Chinese, and the super- vised aligner Nile for Japanese-English (Riesa and Marcu, 2010), with the alignment models being trained on the alignments distributed with the Ky- oto Free Translation Task. 4 To measure translation accuracy, we use the au- tomatic evaluation measures of BLEU (Papineni et al., 2002) and RIBES (Isozaki et al., 2010) mea- sured over all sentences in the test corpus.  We also perform a manual evaluation on 120 sentences from the phrasebook corpus and 80 sentences from the conversation corpus.  These were randomly selected from all sentences of length 1-30, and graded using 1-5 adequacy (Goto et al., 2011) as our evaluation measure.  We also report the percentage of sentences that received a rating of greater than or equal to 2, indicating that the main points of the sentence can be understood, possibly with some difficulty.
Experimental Results
4.2
The results of the experimental evaluation are shown in Figure 3. This graph shows many results, but we first focus on the furthestmost right graph, which shows the percentage of sen- tences understandable to some extent for each of the systems. From this graph, we can see that the scores range from 81% understandable sentences for Japanese-Chinese phrasebook sen- tences, to only 33% understandable sentences on Japanese-English phrasebook sentences.  On the other hand, for conversational sentences, most lan- guage pairs hovered at around 55% understand- able, with Japanese-English being significantly worse. An in-depth analysis of the mistaken sentences identified several issues for improvement that were generally shared by all three systems. Omitted pronouns: Japanese is a pro-drop lan- guage, which means that pronouns, usually the subject of the sentence can be omitted and inferred from the context.  This phenomenon is particularly prevalent in the types of dia- logue contained in the conversation corpus, with the majority of sentences having their subject omitted.  Given that it is difficult for the translation systems used in the experi- ments to accurately reproduce these omitted subjects in a non-pro-drop target language such as English, it is likely that replacing these subjects in a preprocessing step would lead to gains in accuracy (Taira et al., 2012) Dropped words: There were many cases where words central to the sentence were missing from the translation output by the system. This problem is rooted in a number of prob- lems, such as words being mistakenly un- aligned in the training data. Word segmentation:  Both Chinese and Japanese require the segmentation of raw text into words, but occasionally word segmentation errors occurred either due to conversational speech or specialized medical terms.  Thus, using domain adaptation techniques (Neubig et al., 2011) to fix the word segmentations in the medical domain could potentially im- prove down-stream accuracy of translation as well. Medical domain terms:  As expected, there were a few medical domain terms not covered by corpora from more general domains, such as “Benadryl.”  However, the number was also relatively small, with only 5 untranslatable words occurring in a 200 sentence corpus. Overall, an interesting shared point between the majority of members of the list is that they are not
specific to medical translation, but more related to the style of the text. Thus while raising the level of medical MT will certainly involve covering med- ical terminology, it is also equally, if not more, important to overcome obstacles facing the more general speech translation task as well. Finally, in Table 3, we show concrete exam- ples for each of the three translation methods in Japanese-English translation.  The first example is from the phrasebook data, uses some medical terms, and has a very typical syntactic structure for a written Japanese sentence.  As a result F2S is able to translate almost perfectly, but PBMT and Hiero have reordering problems garbling the meaning of the sentence. The second example lit- erally means “other eye, please,” and PBMT is able to generate this very literal translation. Hiero, on the other hand, mistakenly makes the listener the subject of "check," and F2S mistakenly trans- lates “please” as “I’d like,” which doesn’t make sense in this context.  In the third example, all three systems have trouble translating the collo- quial word for “blink one’s eyes,” with PBMT and Hiero dropping the word altogether, and F2S leav- ing it untranslated.
5
Conclusion and Future Work
In this paper, we described an overall design for a speech translation system that aims to reduce the problems caused by language barriers in medical situations.  We describe a collection of a medi- cal corpus, and some translation experiments per- formed on this corpus.  As a result of the experi- ments, we find that the best of three modern trans- lation systems is able to translate 33%-81% of the sentences in a way such that the main content is understandable. While these preliminary results are encourag- ing, this is just the first step towards a full medi- cal speech translation system. As described, there are still a number of challenges related to the MT module itself, including the handling of informal speech.  These will further be compounded when combined with the need for robust ASR and TTS. However, given the potential for speech translation technology to be useful in medical situations, we believe that meeting these research challenges is a worthy target for research and development in the near future.
Acknowledgments