Field Testing the Tongues Speech-to-Speech Machine Translation System

The Tongues portable, rapid-development, speech-to-speech machine translation system was developed specifically to allow a realistic field-test of a deployable prototype. In this paper we will describe the system, its field-testing using regular US Army officers and naive Croatians, and the evaluation of these tests. The evaluation includes analysis of answers to a questionnaire, analysis of system transcript logs, and the authors’ qualitative observations. The overall result of the test was that while the system did successfully aid translation, it requires further development before it would be ready for regular field use. 1. The Tongues System The Tongues system was funded by the US Army to support the mission of the US Army chaplains, who are increasingly called upon to deal with local populations, usually without the benefit of human translators. It is thus intended to be used by a trained US Army chaplain with a completely naive and untrained non-English speaker. The architecture and user interface of the Tongues system were based in large measure on the Diplomat system (Frederking et al., 2000). The speech recognition system used was the open-source Sphinx II (Huang et al., 1992); the translation system was a EBMT/MEMT (ExampleBased MT/Multi-Engine MT) system (Brown, 1996; Frederking and Nirenburg, 1994; Brown and Frederking, 1995) very similar to that in Diplomat; and the synthesis system was the open-source Festival (Black et al., 1998). While the initial system was specifically to demonstrate translation in both directions between English and Croatian, the design was also required to allow rapid development for new languages. To ensure rapid development, the entire project was only allowed to take one calendar year, including contractual arrangements, hiring language experts, etc. The total development effort was similarly restricted: six senior research personnel (the authors of this paper) provided an estimated total of about two (2) fulltime person-years of effort. In addition to the senior staff, there were also part-time Croatian informants, chaplains, and some student programmers. We should note that some of the translation data used to train the system was collected for the Diplomat project (Frederking et al., 2000). In addition to rapid development, the system was not permitted to be restricted to a narrowly-limited domain, but had to be wide-coverage. (Both of these properties were important for the chaplains’ envisioned activities.) Since we were to build a broad-coverage system in a short period of time on a small budget, data-driven approaches were the only reasonable choice. In order to provide in-domain conversational data, we arranged at the start of the project to record a number of chaplains in role-playing conversations of the type they expected the device to encounter. Fortunately, the chaplains were familiar with role-playing exercises, and all had relevant field experiences to re-enact. Both sides of the conversations were spoken in English. These were digitally recorded with head-mounted microphones at 16KHz in stereo (one speaker on each channel), as this was closest to the intended audio channel characteristics of the eventual system. In all, we recorded 46 conversations, ranging from a few minutes to 20 minutes length. This provided a total of 4.25 hours of actual speech. The recorded conversations were hand-transcribed at the word level, and translated into Croatian by native Croatian speakers. The English recordings were used for training the English speech recognition models. The transcripts and their translations were added to the EBMT system’s example base of parallel sentences. A subset of the Croatian translations were read by native Croatian speakers to create data for the Croatian speech recognizer, as described elsewhere (Black et al., 2002). This simple approach appears to be surprisingly adequate. Simply stringing together a recognizer, translator, and synthesizer does not make a very useful speech-to-speech translation system. A good interface is necessary to make the parts work together in such a way that a user can actually derive benefit from it. Using our experience from the earlier Diplomat system, we designed the Tongues interface to be asymmetric, with the Croatian side being as simple as possible, and any necessary complexity handled on the English side, since the chaplain would be trained and practiced in using the system. Even the English side was not terribly complex (see Figure 1). We included a back-translation capability, to allow a user with no knowledge of the target language to better assess the quality of the translation. (We could not use the approach of generating paraphrases from meaning representations, since the system does not use any meaning representations.) We also included several user-requested features, such as built-in pre-recorded instructions and explanations for the Croatian (since the Croatian speaker is completely naive regarding the device and the chaplain’s intentions), emergency key phrases (such as “Don’t move! ”), and enhancements such as being able to modify the translation lexicon in the field, so that the system could be tuned to more specific tasks. The final system ran on a Windows-based Toshiba Libretto, running at 200MHz with 192MB of memory. At the time of the project (2000) this was the best combination of speed and size that was readily available. The system was equipped with a custom touchscreen, so that the Croatian-speaker would not need to type or use a mouse at all. Aware that the system might be used in situations where the non-English participant would be unfamiliar with computer technology, we included a microphone/speaker handset that looks like a conventional telephone handset. This has the advantage of provided a close-talking microphone, thus making speech recognition easier, while coming in a form factor that will be familiar to most people. We have provided a more detailed description of the development of the Tongues system elsewhere (Black et al., 2002). Our design provides abundant opportunities for user error correction, in an effort to enable cooperative users to communicate well enough to accomplish significant tasks that they could not accomplish without the system (or a bilingual human interpreter), despite the error-prone nature of current speech recognition, broad-coverage rapiddevelopment machine translation, and speech synthesis. Determining whether we have met such a goal requires task-based evaluation; while error rates of components are useful information, the real system-level issue is whether communication is achieved, and at what level of effort. Figure 1: Tongues User Interface.