Bilingual Speech Dialogue Corpus for
Simultaneous Machine Interpretation Research
Koichiro Ryu† , Shigeki Matsubara†† , Nobuo Kawaguchi††and Yasuyoshi Inagaki††††
Graduate School of Engineering, Nagoya University††
Information Technology Center/CIAIR, Nagoya University†††
Faculty of Information Science and Technology, Aichi Prefectural University
Furo-cho, Chikusa-ku, Nagoya, 464-8601, Japan
E-mail:k ryu@inagaki.nuie.nagoya-u.ac.jp
Abstract
Speech-to-speech translation has been an important research topic with the advanceof technologies for speech processing and language processing. This paper describes abilingual speech dialogue corpus which has been constructed for research on simultaneous machine interpretation at the Center for Integrated Acoustic Information Research(CIAIR), Nagoya University. The corpus has been implemented by collecting simulated cross-lingual conversations between English speech and Japanese speech throughsimultaneous interpretation, and by transcribing them manually with bilingual sentencealignment. In the year 2002, 216 spoken dialogues have been collected under a realenvironment, and transcribed into text ﬁles consisting of about 300,000 morphemes.
In order to utilize the bilingual corpus eﬀectively, every source utterance speech has
been segmented into interpreting units according to its word-for-word translation andthe word alignment of them. The interpreting unit means a linguistic chunk that could
be interpreted separately and simultaneously. This paper has investigated linguistic
characters of such the unit, and examined the feasibility of simultaneous machine interpretation.

1

Introduction

Speech-to-speech translation has become one of the important research topics with the advanceof technologies for speech processing and language translation. Several experimental systems of
spoken dialogue translation for speciﬁc task domains have been developed so far (Takezawa etal., 1998) (Watanabe et al., 2000). The interpreting style of them is within so-called consecutive
interpretation. In order to provide an environment for supporting natural and smooth crosslingual communication, it is desired to develop a simultaneous interpretation system. Toward
simultaneous machine interpretation, not only the quality of the interpretation but its output
timing is also important, and it would be eﬀective to investigate and analyze the interpreting
process of professional simultaneous interpreters.
This paper describes a bilingual speech dialogue corpus, which has been collected as part of the
project on the massive-scale speech database construction in the Center for Integrated Acoustic
Information Research (CIAIR) (Kawaguchi et al., 2002). The bilingual spoken dialogues between
an English native speaker and a Japanese native speaker through professional simultaneousinterpreters have been collected and transcribed into ASCII text ﬁles. The exact beginning timeand end time are provided for each utterance, and moreover, each source utterance is alignedwith the corresponding target utterance. In this year, 216 spoken dialogues have been collected
in total.
In order to utilize the corpus for the research on simultaneous machine interpretation, every
speaker’s utterance is segmented into “interpreting unit” The segmentation is executed basedon the word-for-word translations for spoken source sentences and the word alignments of them.
The interpreting unit means the smallest speech unit which can be translated separately andsimultaneously. Therefore, it is very important for a simultaneous interpretation system toFigure 1: Recording scene (Upper left: English speaker, Upper right: Japanese speaker, Lower
left: English-Japanese interpreter, and Lower right: Japanese-English interpreter)recognize such a unit. In this paper, we make an investigation into a possibility of simultaneousmachine interpretation by extracting such the units from our bilingual corpus.
This paper is organized as follows: Section 2 explains the design, construction and statistics
of our corpus. Section 3 describes the segmentation of source utterances into interpreting units.
Section 4 provides the observations of the interpreting unit.

2

Simultaneous Interpreting Corpus

Several bilingual spoken dialogue corpora have been constructed, and played important roles in
the advancement of the speech translation technology (Ehara et al., 1990; Morimoto and et al.,1994). Most of them, however, are limited to the collection of consecutive interpreting data,
and therefore, simultaneous interpreting corpora will be getting valuable in the coming machineinterpretation research.
The various types of large-scale multilingual speech database have been constructed at CIAIR
from 1999 (Aizawa et al., 2000; Matsubara et al., 2002). The bilingual speech dialogues between
an English speaker and a Japanese speaker through simultaneous interpreters were collected in
the year 2002. This section explains the recording, transcription, and size of our spoken dialogue
corpus.
2.1 Data Collection
We have set up “airport” and “hotel” as the typical situations of dialogue communications inan overseas travel. To put it concretely, the following topics were selected: “airport checkin”, “hotel check-in/check-out”, “booking of a room at a hotel”, and “booking of a seat in an
airplane”, and so on. Since it is diﬃcult for the subjects to talk naturally and smoothly withno motivation, we prepared the dialogue speciﬁcation sheets and showed the subjects them inadvance.
Because CIAIR has a research purpose of collecting large-scale speech data under real environments, and therefore our recording was carried out in a classroom. The sounds of native speakers
and the interpreters were digitized by 16kHz of sampling frequencies and 16 bits, and recorded
in stereo onto digital audio tapes (DAT). We have recorded 216 dialogues in total. Generally
simultaneous interpreters use not only speaker’s utterances but also speaker’s facial expressions
and conversational behaviors. So simultaneous interpreters enter a booth, where he or she canlook speakers over glass window, when our recording was carried out(Figure 1). In order to enhance the quality of the interpretations for the English speaker and the Japanese speaker, eachspeaker was accompanied by one interpreter. To ensure all the participants the speakers can only
listen to the output from the other speaker’s interpreter, and the interpreter can only listen toFigure 2: Sample of the transcript
Table 1: Basic statistics of the simultaneous interpreting corpus
item
recording time(minute)speaking time(sec)morphemes(words)
kinds of morpheme(word)utterance units
sentence breaks
ﬁllersEnglish
1082
17865
73513
2713
10858
7889
2376

E-J
1082
17495
77790
2855
11354
9595
4396

Japanese
1082
18534
75064
2601
13176
8725
4892

J-E
1082
17128
64306
2615
10813
6991
2712

the speech that they are assigned to interpret. Please note that this is a simulative dialogue, inwhich the contents of the utterances can be limited. Therefore, such background information as
the speakers’ roles and the dialogue tasks were informed to the speakers in advance, in attempt
to collect as spontaneous an utterance as possible.
2.2 Speech Transcription
The collected speech data was transcribed into the text manually. The text transcription
was done according to the manual for Corpus of Spoken Japanese(CSJ) produced by National
Japanese Language Research Institute (Maekawa et al., 2000). Figure 2 shows the sample ofthe transcript. The dialogue tags were provided for the language phenomena characteristicof spoken language, such as ﬁllers(F), corrections(X), hesitations(D), mistatements(W), repetitions(S), monologues(L) and sentence breaks SB. Moreover, the transcript is broken up intoutterance units by the pauses for 200ms or more, or those for 50ms after the sentence break,and the beginning time and end time have been provided for every utterance unit.
Moreover, the alignment between Japanese speaker’s utterances and Japanese-English interpreter’s utterances, and that between English speaker’s utterances and English-Japanese interpreter’s utterance have been provided. Figure 2 shows the sample of the alignment.
2.3 Statistics of the Corpus
The basic statistics of our corpus, such as recording time, speaking time, number of morphemes(words), diﬀerent morphemes, utterance units, sentences, and ﬁllers, are shown in Table 1. E-J
and J-E in the table indicate English-Japanese interpreter’s and Japanese-English interpreter’s,
respectively. In this paper, a morpheme in English means a word, and the number of morphemesin Japanese was calculated on the basis of the result of a Japanese morphological analyzer called
ChaSen (Matsumoto and et al., 2000). The number of kinds of morpheme in English is theFigure 3: Pattern diagram of simultaneous interpretation
Table 2: Statistics of the aligned corpus
item
overlapping time(sec) (1)diﬀerence of beginning time(sec) (2)diﬀerence of end time(sec) (3)speaker’s speaking time(sec) (4)interpreter’s speaking time(sec) (5)speaker’s morphemes(words)
interpreter’s morphemes(words)

E-J
0.63
1.68
1.81
2.19
2.20
9.47
10.29

J-E
0.65
1.88
1.78
2.21
2.15
9.33
8.49

number of words whose notations diﬀer, and that in Japanese is the number of morphemes
whose basic forms diﬀer. The recording time is about 18 hours. The total of the speaking time
is about 20 hours data. The reason for that the total of the speaking time is longer than that of
the recording time is that the utterance of the interpreter overlaps with the speaker’s utterance.
As the result of the parallel corpus alignment, the average time of several indices shown in
Figure 3 has been measured.
(1) The time overlapping with the speaker’s utterance in the interpreter’s utterance.
(2) The diﬀerence between the beginning time of the speaker’s utterance and that of the interpreter’s one.
(3) The diﬀerence between the end time of the speaker’s utterance and that of the interpreter’sone.
(4) The speaking time by speakers.
(5) The speaking time by interpreters.
In addition, the average number of the morphemes of speaker’s utterances and interpreter’sutterances has been also counted. Table 2 shows the statistics of the aligned corpus.

3

Speech Segmentation into Interpreting Units

The biggest diﬀerence between consecutive interpretation and simultaneous interpretation would
be the beginning time of the interpretation. The simultaneous interpreter begins to speak before
the source utterance ﬁnishes, and that enables the simultaneous interpreter to reduce listener’s
waiting time and make a contribution to natural bilingual conversations. In general, simultaneous interpreters break up the utterance by speakers into several meaningful segments, andtranslate them incrementally. We call such the segment (simultaneous) interpreting unit. If a simultaneous interpretation system can recognize such the unit, it can begin to make a translation
without waiting for the completion of the speaker’s utterance. Moreover, there is an advantagethat the output staring timing can be decided inevitably. As one of the techniques for acquiring
the interpreting units, a group of the meaning is formed by referring to the correspondence of thewords of the original and the interpreted sentence. That is, the group becomes an interpretingunit. However, it is diﬃcult to use simultaneous interpreter’s transcript data to acquire theinterpreting unit. Since simultaneous interpreters use advanced peculiar techniques such as thefree translation, the omissions, and restating to overcome a time restriction, it does not reﬂect
the sentence structure of the speaker’s utterance in precision. Therefore, in our research, theword-for-word translations are given to the transcript of the speaker’s utterance manually without putting time pressure. These are the interpretations based on the structure of the speaker’s
utterance. That is, they are interpretations with little free translation and omission. And, thegroup of the meaning is found to the speaker’s utterance by refering the word correspondence
between the word-for-word translation and the speaker’s utterance. Interpreting unit can beacquired by breaking up the speaker’s utterance into the unit.
3.1 Word-for-Word Translation
The word-for-word translations are given to the transcript of the speaker’s utterance manually.
To collect data with good quality that aimed at the construction of the simultaneous machineinterpretation, the person who had simultaneous interpreter’s expertise did this work. The
method of making the word-for-word translation is described in detail as follows.
Rules of word-for-word translation:
1. Each utterance is given the word-for-word translation. (Exceptionally, it is likely to extend
two or more utterances only when not interpretable by one utterance alone. )2. The interpreter’s utterances in the simultaneous interpreter corpus are not referred to.
3. The listener can understand the semantic content of the original utterance exactly.
4. The system can translate the source utterance incrementally.
5. To avoid the dependence on the context, the free translation and the omission are avoided.
Here, the example of the word-for-word translation is shown (Figure 4). The beginning utterance ID and the end utterance ID of the corresponding speaker’s utterance are given to eachtranslation. In Figure 2, the interpreter doesn’t translate “I’m afraid” that exists in the original. (It is guessed that there was no translated time.) Moreover, the interpreter has translated
the part of “very diﬃcult” in “very diﬃcult to get a ﬂight” freely with “komi atte iru (It is
crowded).” In word-for-word translation “I’m afraid” in the speaker’s sentence is interpreted
like ”moushiwake gozaimasenn (I’m sorry).” And the part of “very diﬃcult” reﬂects speaker’s
sentence like “sore-ha muzukashi-i (It’s very diﬃcult).” The omission and the free translationinterferes to the translation making when using it as study data of the simultaneous machineinterpretation of the example base. Therefore, making the word-for-word translation withoutomissions and free translations is proﬁtable.
3.2 Acquirement of Interpreting Units
This section describes the technique for acquiring the interpreting unit by using the word-forword translation made with section 3.1. We explain the procedure of the unit division by using
Figure 5.
Step 1 We break up English utterance/interpretation into words (Japanese utterance/interpretation
into morphemes).
Step 2 The word correspondence is taken between speaker’s utterances and interpretation. In
Figure 5 an arrow show the correspondence.
Step 3 Then, sometimes, the phenomenon of arrows intersecting occurs because of the diﬀerenceof the grammar structure between English and Japanese. In Figure 5 the intersection of thearrow exists. It is caused by the diﬀerence of grammar that a verb is located just behindthe subject in English and it is located in the end of sentence in Japanese. The part where
the arrow intersects settles the two as one interpreting unit. Therefore, it is possible totranslate natural interpretation without disregarding the grammar.

(a) English native speaker(b) English-Japanese interpretation

Figure 4: Sample of the transcript (word-for-word translation)

Figure 5: Example of segmentation of interpreting unit

4

Investigation of Interpreting Unit

4.1 Statistics of Interpreting Unit
As the basic statistics of word-for-word translation data, the number of morphemes (words),
utterance units, sentences, and turns. are shown in Table 3. The turn is a group of one or moreutterances until a speaker alternates. And Table 4 shows the number of interpreting units per autterance, number of interpreting units per a sentence, number of interpreting units per a turn,and number of morphemes (words) per a unit. According to Table 4, it is possible to segment aturn into the unit of 3.67 in English and 3.73 in Japanese on average. It means that the chanceto output the interpretation increases compared with before speaker’s utterances are segmented.
4.2 Utilization of Interpreting Unit
It is explained that the process of acquiring the interpreting unit from speaker’s utterances in
section 3.2. Acquiring this interpreting units may give the possibility that the simultaneousmachine interpretation can imitate simultaneous interpreter’s behavior. Figure 6 shows the
distribution of the number of morphemes in the ﬁrst unit of a turn. The units composed less thanfour words accounts for 90% or more in English and less than six morphemes accounts for 90%
or more in Japanese. That is, if the interpretation is output incrementally in each interpreting
unit, the system begins the interpretation before the speaker ﬁnishes speaking. Therefore it
can reduce listener’s waiting time, and a conversation between diﬀerent languages be achieved

Table 3: Basic statistics of the word-for-word translation
item
EnglishE-J JapaneseJ-Emorphemes(words)
69424 68675
70225 62605
utterance units (translations)
10858 10387
13176 12516
sentence breaks
7889
8108
8725
8231
turns
4451
4451
4422
4422

Table 4: Basic statistics of interpreting unit
item
interpreting units
average units per a utterance
average units per a sentence
average units per a turnaverage morphemes per a unit

E-J
16338
1.50
2.07
3.67
4.25

J-E
16516
1.25
1.89
3.73
4.25

smoothly. Figure 7 and 8 show the position where the unit boundary in the sentence appearedat the rate. This indicates the possibility that the translations can be output incrementally
along a speaker’s utterance if an interpreting unit is used. Figure 9 shows the distribution of
the number of morphemes per a unit. About 80% of all units is composed less than ﬁve words
in English and less than 7 morphemes in Japanese. Thus, being possible to segment into a
short unit is that there is a chance to output interpretation. If these chances are used, listener’s
waiting time is reduced and the interpreting system can output interpretation incrementally.

5

Concluding Remarks

This paper has described the design of a bilingual speech dialogue corpus, which has been constructed at the CIAIR, and its investigation. Our corpus contains spoken dialogues between
an English native speaker and a Japanese native speaker through professional simultaneousinterpreters, and each source utterance has been aligned with the corresponding target utterance. In order to utilize the corpus for the research on simultaneous machine interpretation,
every speaker’s utterance is segmented into interpreting units. The segmentation is executed
according to word-for-word translation for spoken source sentences and alignment between thecorresponding words. The results of the investigation of interpreting units are as follows:• It is possible to segment into the unit of 3.67 in English and 3.73 in Japanese averages aturn.
• The ﬁrst unit of turn composed less than four words accounts for 90% or more in English
and less than six morphemes accounts for 90% or more in Japanese.
• About 80% of all units is composed less than ﬁve words in English and less than 7 morphemesin Japanese.
The results would provide the possibility that a simultaneous interpreting system can interpretsource languages incrementally and play a role for supporting natural and smooth cross-lingual
communication.

Acknowledgements
Inter Group Corporation has cooperated in the recording of the data. Many thanks to Mr.
Masafumi Yokoo for his contribution. The authors would like to express their gratitude to Ms.
Yayoi Banno and Ms. Nozomi Yamaguchi for their contribution in constructing the interpreting
unit data. This research is partially aided by the Grant-in-Aid for COE Research for Young
Scientists of the Ministry of Education, Science, Sports, and Culture, Japan and the Nakajima
Foundation.

EJ
JE

2500
2000
1500
1000
500
0
0

2

4

6

8

10 12 14 16 18 20

number of interpreting units

number of interpreting units

3000

2000
1800
1600
1400
1200
1000
800
600
400
200
0

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1number of morphemes from sentence beginning/ number of morphemes of sentence

number of morphemes

1800
1600
1400
1200
1000
800
600
400
200
0

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

1number of morphemes from sentence beginning/ number of morphemes of sentence

number of interpreting units

number of interpreting units

Figure 6: Distribution of number of mor- Figure 7: Distribution of interpreting units
pheme in ﬁrst unit
(English-Japanese)
7000

EJ
JE

6000
5000
4000
3000
2000
1000
0
0

2

4

6

8

10 12 14 16 18 20

number of morphemes

Figure 8: Distribution of interpreting units Figure 9: Distribution of number of mor(Japanese-English)
phemes per an interpreting unit

References
Aizawa, Y., S. Matsubara, N. Kawaguchi, K. Toyama, and Y. Inagaki. 2000. Spoken Lauguage Corpus for
Machine Interpretation Research. Proceedings of 6th International Conference of on Spoken Language
Processing, III:389–401.
Ehara, T., K. Ogura, and T. Morimoto. 1990. ATR Dialogue Database. Proceedings of 1st International
Conference of on Spoken Language Processing, pages 1093–1096.
Kawaguchi, N., S. Matsubara, K. Takeda, and F. Itakura. 2002. Multi-Dimensional Data Acquisition
for Integrated Acoustic Information Research. Proceedings of International Language Resources and
Evaluation Conference, pages 2043–2046.
Maekawa, K., H. Koiso, S. Furui, and H.Isahara. 2000. Spontaneous Speech Corpus of Japanese. Proceedings of 2nd International Conference of Language Resources and Evaluation, pages 945–952.
Matsubara, S., A. Takagi, N. Kawaguchi, and Y. Inagaki. 2002. Bilingual Spoken Monologue Corpus for
Simultaneous Machine Interpretation Research. Proceedings of International Language Resources and
Evaluation Conference, I:153–159.
Matsumoto, Y. and et al., 2000. Morphological Analysis System ChaSen version 2.2.1 Manual.
Morimoto, T. and et al. 1994. Speech and Language Database for Speech Translation Research. Proceedings of 3rd International Conference of on Spoken Language Processing, pages 1791–1794.
Takezawa, T., T. Morimoto, Y. Sagisaka, N. Cambell, H. Iidaand F. Sugaya, A. Yokoo, and S. Yamamoto. 1998. Japanese-to-English Speech Translation System: ATR-MATRIX. Proceedings of 5thInternational Conference on Spoken Language Processing, pages 957–960.
Watanabe, T., A. Okumura, S. Sakai, K. Yamabana, S. Doi, and K. Hanazawa. 2000. An AutomaticInterpretation System for Travel Conversation. Proceedings of 6th International Conference on Spoken
Language Processing, pages IV:444–447.