End-to-end evaluation of a speech-to-speech translation system in TC-STAR

The paper describes an evaluation methodology to evaluate speech-to-speech translation systems and their results. The evaluation scheme uses questionnaires filled in by human judges for addressing the adequacy and fluency of audio translation outputs and was applied in the second TC-STAR evaluation campaign. The same evaluation methodology is carried out both on the outputs of an automatic system and on human translations produced by professional interpreters. The obtained results show that the professional interpreters obtain better results on fluency. But surprisingly, the automatic systems results on adequacy are higher than for the human translator. But this has to be reconsidered by the fact that humans have to translate in a real time, and select important information, while an automatic system tends to translate all the information. Introduction The TC-STAR project is a long-term effort to advance research in the core technologies of Speech-to-Speech Translation (SST). The project targets a selection of speech domains (speeches and broadcast news) and three languages: UK English, European Spanish and Mandarin Chinese. To assess the advances made in all SST technologies, annual competitive evaluations are organized. In the second evaluation campaign of TCSTAR which took place in February and March 2006, an end-to-end evaluation was carried out. A Speech-to-Speech Translation system is composed of an Automatic Speech Recognizer (ASR) chained to a Spoken Language Translation (SLT) module and to a Text-To-Speech (TTS) component in order to produce the speech in the target language. Evaluations of individual components (ASR, SLT and TTS) have been done many times in the past and are also evaluated in TC-STAR. Methods, protocols and metrics for the evaluation of independent ASR, SLT and TTS modules have already been established. Performance of speech recognition systems is typically described in terms of word error rate (WER). For SLT automatic metrics such as BLEU (Papineni et al. 2001) or mWER (Nießen et al., 2000) are used. Performance of SLT systems can also be measured by human judges who can usually measure fluency, adequacy, etc. of the translations. A numerical indication of the perceived quality of TTS synthesized audio data is provided by Mean Opinion Scores (ITU-T, 1993). The MOS is generated by averaging the results of a set of standard subjective tests. To evaluate the performance of a complete speech-tospeech translation system, we need to compare the source speech used as input to the translated output speech in the target language. The proposed methodology enables to measure the fluency and the adequacy of the translated output. We first give a quick overview of the TC-STAR evaluation results. Then we describe tasks and languages of the experiments, and the end-to-end evaluation methodology and protocol. Finally, the results obtained by the TC-STAR system and the human interpreter are depicted and analyzed. 1 http://www.tc-star.org Overview of the TC-STAR evaluation results In this section we introduce the TC-STAR evaluation results regarding the individual component (ASR, SLT, TTS) evaluated within the project. It aims at defining the scope of the end-to-end evaluation through the state-ofthe-art of each component evaluation. Well-know evaluations are used and details can be found in (Mostefa et al., 2006). Three languages are involved by using recordings of the Parliament’s sessions in English and Spanish, recordings of Spanish Parliament’s sessions in Spanish and broadcast news recordings in Chinese. Automatic Speech Recognition Evaluation The results in term of Word Error Rate (WER) for English and Spanish and Character Error Rate (CER) for Chinese are shown in Table 1. English (WER) Spanish (WER) Chinese (CER) Top 1 8.2% 10.2% Tc-star 6.9% 8.1% 9.8% Table 1: ASR results in WER/CER For English, the best results are obtained by the TC-STAR system combination with a Word Error Rate of 6.9%, while the best individual system has obtained a WER of 8.2%. For Spanish, the best performance of an individual system is 10.2%. Again a ROVER combination of all hypotheses is performed and gives the best results with a WER of 8.1%. For Chinese, a common submission from two sites has performed with a CER of 9.8%. Spoken Language Translation Evaluation Three kinds of text data are used as input: • The output of the automatic speech recognition systems (ASR); • The manual transcriptions (Verb.) including spontaneous speech phenomena, such as corrections, false-starts, etc.; • The Final Text Editions (FTE) provided by the European Parliament. Some sentences are rewritten and text data do not include transcription of spontaneous speech phenomena. The results in term of BLEU scores (Papineni et al. 2001) for English-to-Spanish (EnEs), Spanish-to-English (EsEn) and Chinese-to-English (ZhEn) are shown in Table 2. EnEs EsEn ZhEn ASR Verb. FTE ASR Verb. FTE ASR Verb. Top 1 35.97 46.61 49.81 39.41 52.54 48.1616. 07 12.39 Tc-star 47.53 50.74 52.55 48.07 Table 2: BLEU scores [%] for the SLT evaluation The SLT ROVER combination of automatic translation has been performed in addition to individual systems outputs and achieves the best results for most of the tasks. Moreover, a human evaluation has been carried out for the Spanish-to-English direction in which each segment has been evaluated in relation to both adequacy and fluency measures (White et al. 1994). Two evaluations per segment were carried out by 130 native Spanish evaluators. A total of 20,360 segments were evaluated. Table 3 shows the results of the SLT human evaluation for the Top 1 system, the ROVER and also for one of the two reference translations used for the automatic evaluation. Top 1 ROVER Human Ref. Task ASR Verb. FTE Verb. FTE Verb. FTE Fluency 3.06 3.39 3.63 3.32 3.46 4.31 4.56 Adequacy 3.13 3.54 3.79 3.55 3.72 4.31 4.44 Table 3: Fluency and adequacy judgements on a scale from 1 to 5 (5: best; 1: worst) Regarding the general performance of the automatic system, results are good, as the scores are all above 3. However, the difference between the human reference and the automatic systems is still considerable. The ROVER system is not better than the Top 1 system although the scores are very close. Text to Speech Evaluation TTS evaluation is made of subjective tests to evaluate the prosody, the acoustic synthesis, the expressive speech, the voice conversion and the global TTS component, done by 20 native speakers. Table 4 presents the results for the overall quality test of the TTS component on the English, Spanish and Chinese languages. The evaluation is done in a translation scenario, using the ASR and SLT outputs. English Spanish Chinese Top 1 3.41 4.33 3.84 Natural Voice 4.79 4.66 4.44 Table 4: Overall quality (1-5) for TTS evaluation Natural voice obtains the best performance compared to the best TC-STAR system. However, for Spanish, the best system is pretty close to the natural voice. In the following sections we describe the end-to-end procedure and results. End-to-End Evaluation