Construction and Utilization of Bilingual Speech Corpus for
Simultaneous Machine Interpretation ResearchHitomi Tohyama†, Shigeki Matsubara‡, Nobuo Kawaguchi‡,and Yasuyoshi Inagaki∗†Graduate School of Information Science, Nagoya University, Japan‡Information Technology Center, Nagoya University, Japan∗Faculty of Information Science and Technology, Aichi Prefectural University, Japan
hitomi@el.itc.nagoya-u.ac.jp

Abstract
This paper describes the design, analysis and utilization of asimultaneous interpretation corpus. The corpus has been constructed at the Center for Integrated Acoustic Information Research (CIAIR) of Nagoya University in order to promote therealization of the multi-lingual communication supporting environment. The size of transcribed data is about 1 million words,
and the corpus would deserve to be called the simultaneous interpretation corpus of the largest-in-the-world class. The discourse tag and the utterance time tag were given to the corpus,
and some software tools for corpus analysis in order to support
the practical use of the corpus have been developed. Therefore,
the corpus is expected to be useful not only for the developmentof simultaneous interpreting systems but also for the construction of an interpreting theory.

1. Introduction
Recently, spoken language corpora have been constructed forthe purpose of studying on speech processing in many researchorganizations (for example [1, 2]). The large-scale corpora arerecognized to be important widely, and used in various researchareas, such as speech recognition, natural language processing, linguistics, language education, and dictionary compilation. At the Center for Integrated Acoustic Information Research of Nagoya University (following, CIAIR), a corpus ofsimultaneous interpretation between Japanese and English has
been constructed over five years. We aim at the realization of
the multi-lingual communication supporting environment. The
recording time is 182 hours in total. The speech data has been
all transcribed and visualized. Furthermore, we have completed
language analysis of the corpus. The size of transcribed data
is about 1 million words, and the corpus would deserve to becalled the simultaneous interpretation corpus of the largest-inthe-world class. Additionally, we have developed some software tools for corpus analysis in order to support the practicaluse of the corpus. They have been developed as software which
can be performed on the Web server, and a user can refer to thecorpus easily by using a browser.
This paper describes the design, collection, construction,
analysis, and utilization of the simultaneous interpretation corpus. In the following section, we describe the purpose and thedesign of the simultaneous interpretation corpus. In Section 3,
we describe the recording of the corpus. In Section 4, the construction of the corpus is explained in full detail. Section 5 discusses the use of the corpus.

speakerinterpreter

Figure 1: Recording environment

2. Design of the Corpus
2.1. Aim of Data Collection
Machine interpretation has become one of the most importantresearch topics with the advance of technologies for speech processing and language translation. Several experimental systemsof spoken dialogue translation for specific task domains havebeen developed [3, 4, 5]. The interpreting style of them is within
so-called consecutive interpretation. In order to provide an environment to support natural and smooth cross-lingual communication, to develop a technique for simultaneous machine interpretation has been awaited and tried out recently. Not only
a generation of translation but an outputting timing of translation is required for a simultaneous interpreting system. It
would be effective to investigate and analyze the interpreting
process of professional simultaneous interpreters. The CIAIR
is constructing and maintaining various types of speech andlanguage database for the purpose of the advancement of robust speech information processing technology [6]. Moreover, abilingual database of simultaneous interpretation has also been
constructed as a part of this project. We aim to develop speechtranslation technologies and to construct interpreting theories.
2.2. Policy of Corpus Design
The large-scale corpus needs to be equipped with flexibility, because many researchers are expecting to utilize the corpus fortheir purposes. Therefore, we collected both monologue and dialogue data. The contents of the database are daily topics. The
database targets English and Japanese.

3. Recording of Speech Data
3.1. Recording Environment
One of the purposes of CIAIR is to collect a large quantity ofspeech data which were generated under natural circumstances,
so the recording took place in a classroom. Facial expressions
and conversational behaviors of speaker’s are also important information for simultaneous interpretation, thus, the interpreterswere in booths from which they could see the speakers as Fig.1
shows. Both speakers and interpreters used the same crosstalking microphones. The speeches were digitized by sampling
frequencies of 16 kHz and 16 bits, and recorded onto digitalaudio tapes (DAT) in multiple channel environments. All interpreters are professional interpreters who are active in the frontlines.

Figure 2: Sample of the transcribed text: English speaker’s utterance

3.2. Recording of Monologue Speech Data
Simultaneous interpreters go into a booth, and interpret the lecturer’s speeches from the headphone. Although the lecturers
face to the audience, they cannot hear the interpreter’ speech.
It enables them to speak at their own paces. The contents ofspeeches are economics, history, and, culture, etc. Moreover,each monologue speech is interpreted by two or four professional interpreters. Their degree of experience differs from one
another (5 years over or not). The flexibility of a database israised by using four interpreters. Therefore, it becomes possible to compare two or more interpretation examples in a sample of utterance. Moreover, we can compare interpreters’ utterance speed, speaking timing, and strategy of interpretation. The
speech was recorded for about 10 minutes per lecture.
3.3. Recording of Dialogue Speech Data
Travel conversation was selected as a domain of conversation,
which includes popular topics during overseas travel at airports
and hotels, and simulated conversations are recorded. To put it
concretely, the following topics were selected: ”airport checkin”, ”hotel check-in/check-out”, ”booking of a room at a hotel”,
and ”booking of a seat in an airplane”, and so on. In order toenhance the quality of interpretations for both English speakers
and Japanese speakers, each speaker was accompanied by one
interpreter. To ensure all the participants’ speech pretension,speakers can listen only to the output from the other speaker’s
interpreter, and the interpreters can listen only to the speech thatthey are assigned to interpret. Please note that these dialogues
are simulative, in which the contents of the speeches can belimited. In attempting to collect utterances as unfettered as possible, such background information as the speaker’s roles andconversational tasks were informed to speakers in advance. Fora speaker who is a customer of a hotel, for example, the kindand number of rooms that should be reserved, and for a speakeras front desk clerk, rooms that can be reserved, etc. We set up”airport” and ”hotel” as the typical situations of dialogue communications doing overseas travel. The recoding time per one
dialogue was from 1 minute to 16 minites, and dialogues of various types were collected.

4. Construction of the Corpus
4.1. Transcription of Speech Data
The transcription was produced based on the standard transcription rules of t he Corpus of Spoken Japanese (CSJ) developed bythe National Japanese Language Research Institute [7]. Figure 2

Figure 3: Sample of the transcribed text: English-Japanese interpreter’s utterance

and 3 show the sample data of English speaker’s utterances andthat of English-Japanese interpreter’s utterances, respectively.
All the speech data (182 hours) were transcribed into a text.
The standardization is shown as follows:• Utterance unit
Utterance units were set by 200ms-or-longer pauses inthe speech of speakers and interpreters.
• Notation
Recorded Japanese speech is transcribed in two different
ways: orthographic and phonetic transcriptions.
• Tag annotation– Utterance ID
A serial number was given to each utterance unit.
– Time tag
The beginning time and end time of the utterance
units were tagged.
– Discourse tag
Language tags were also added onto fillers, hesitations, and corrections.
4.2. Visualization of Speech Data
We developed a timing information visualization tool. The
speaking time of English lecturers, English-Japanese interpreters, Japanese lecturers and Japanese-English interpreters
and their overlapping relations are displayed as Fig.4 shows.

Table 1: Statistics of monologue data
itemEnglishJapaneseTotal
EJ
interpreter JETotal
Sum total
speakerdata size(words)
90249
84278
174527266050127991394041568568

data size(utterances)
8422
6529
14951
25507
16083
41590
56541

recordingtime (min.)
695
597
1292
1639
1265
2904
4196

Table 2: Statistics of dialogue data
item
EnglishJapanese
Total
EJ
interpreter JE
Total
Sum total
speakerdata size(words)10785010625821410811677691743
208519422627

data size(utterances)
14223
16485
30708
15286
13719
29005
59713

recordingtime (min.)
1678
1678
3356
1678
1678
3356
6712

Figure 4: Visualization of the binlingual speech
English speaker

English-Japanese interpreter

Thereby, we can visually observe a overlap of a lecturer and aninterpreter utterance.
4.3. Alignment of the Bilingual Speech
For a detailed analysis of interpreters’ speech, such as extraction of temporal characteristics of interpretation, the acquisitionof translation patterns, the detection of translation units and soforth, it is necessary to align utterances of speakers and interpreters with relatively small units. An alignment support toolworking on the internet, which Fig.5 shows, has been developed by using CGI script. The users can align the utterance
units by the clicking the mouse on the bilingual text displays.
The aligned data can be used for analysis of interpreting unitsand timing. We have aligned the corpus using the tool according
to the following conditions:• The unit should be the minimum alignment unit
• Utterances should be aligned as small as possible• Utterance units such as fillers or non-language phenomena and the utterances with no appropriate counterparts
can have no correspondence.
As stated above, a detailed alignment standard was established,so, the annotation work was made uniform.
4.4. Environment Information
A large-scale corpus has various availabilities. Information that
doesn’t appear at speech data and text data might be important. Therefore, we provided the Environment information ofrecording data file for every session. Environment information
includes date, location, recording time, audio-video equipment,topic, type of speech, the speaker’s roles and conversationaltasks, the information on speaker, the information on interpreter(years of experience, etc).

Figure 5: Alignment of the binlingual speech

4.5. Statistics on the Corpus
The large-scale corpus involving 1 million words have been developed; we have finished recording the data of 182 hours ofspeech, transcribing it into text, attaching discourse tags, andmatching source utterances to their target utterances, so far.

5. Utilization of the Corpus
To develop the simultaneous machine interpretation, it is necessary to analyze the interpreting process of professional simultaneous interpreters. We have researched simultaneous interpreters’ utterance timing, interpretation unit, and generation oftranslation and compared the simultaneous interpretation withthe consecutive one in cross-lingual communication. We haveproved the effectiveness of simultaneous interpretation technology. Furthermore, if the data for analysis is large-scale, it is also
possible to verify a qualitative analysis result still more a quantitative one. This section describes the main research results
performed by using the corpus.
5.1. Analysis of Interpreter’s Speaking Timing
Simultaneous interpretation may overlap with the corresponding native speech. It is expected that an interpreter recognizes

a part of a lecturer’s utterance as an interpreting unit and interprets it at an early stage. We have investigated the interpretingunits and speaking timing of professional interpreter by analyzing the aligned corpus. The summary of results is shown below:• Since a subject appears at the beginning of a sentence inboth Japanese and English, the subject can be interpreted
immediately.
• By controlling the out putting speed of system based on
the quantity of the input utterance. It is possible to reduce the difference between the beginning time of theinterpreter’s utterance and that of the lecturer’s utterance.
5.2. Analysis of Temporal Features
Interpretation has two styles: consecutive interpretation and simultaneous interpretation. One of major differences between
them is whether an interpreter starts to speak after the speakercompleted his/her utterances, which is consective interpretation,or before, which is simulteneous interpretaion [8]. Simulteneous interpretaion occurs that the speaker’s utterance and interpreter’s utterance temporally overlap each other; however inthe consective interpretaion, those utterances doesn’t. We havedone the further research on these two styles of interpretation.
We have compared simultaneous interpretation with consecutive one in cross-lingual communication and we haveproved the possibility that simultaneous interpretation technology performs more effectively. Our study proved how effective
the simultaneous interpretation is by analyzing the actual simultaneous interpretation data. It focused the on the efficiency andthe smoothness of cross-language conversation through simultaneous interpretation. The summary of results is shown below:• The growth of average dialogue time growth rate on simultaneous interpretation was twice as much as consecutive interpretation, which indicates that conversational efficiency through simultaneous interpretation has
been raised considerably in comparison with consecutive interpretation. This tendency can be seen in EnglishJapanese interpretation.
• The average of speakers’ waiting time on conversations
through interpreting is 4.4 seconds for English speakers
on simultaneous interpretation, and 15.4 seconds on consecutive interpretation. That is 4.3 seconds for Japanese
speakers on simultaneous interpretation, and 14.5 onconsecutive interpretation, which indicates that smoothness increases drastically.
The result proved the usability of simultaneous interpreting
technology as a support environment for cross-language dialogues because in the dialogues through simultaneous interpretation.
5.3. Collection of Interpretation Strategies
Simultaneous interpretation is advanced language processing
activities of human. Simultaneous interpreters have to generate their translations simultaneously with original speech.
However, they have the restrictions on speaking timing (whento-say) and how to translate speaker’s utterance (how-to-say).
However, they have various kinds of strategies to raise simultaneity. In this investigation, the interpreting patterns used frequently and having both/either high flexibility and simultaneity were extracted from a bilingual spoken monologue corpus.
The CIAIR corpus has as many of four interpreter data per one
monologue section. Therefore, it is possible to collect two

or more interpretation patterns from one speaker’s utterance.
Those extracted interpretation patterns can be used as a rule ofthe interpretation system. It is possible to develop the system
by using the rule.

6. Conclusion
This paper has described the design, analysis and utilization of
the CIAIR simultaneous interpretation corpus of Nagoya University. We expected that the corpus will be used for not only thedevelopment of simultaneous interpreting systems but also theconstruction of an interpreting theory. We are going to develop
the CIAIR corpus further by giving it more detail tags and constructing alignment data. In those days, the spoken language
technology has progressed. Therefore, the demand for largescale spoken database is rising in not only speech processing
but also cognitive science, phonology, and linguistics. We hopethat CIAIR corpus will be used in the various research fields.
It is preferable to exchange the opinion between different areas
and to progress overall.
The CIAIR corpus has been already exhibited. For moredetails, please refer to the following URL:
http://slp.el.itc.nagoya-u.ac.jp/sidb/

7. Acknowledgments
The authors would like to thank Prof. Dr. Toyohide Watanabe
and all members of EL project for their precious advices. This
work is partially supported by the Grand-in-Aid for Exploratory
Research (No. 17652040) of JSPS.

8. References[1] Maekawa, K., Koiso, H., Furui, S. and Isahara, H., “Spontaneous Speech Corpus of Japanese,” Proc. of 2nd International Conference on Language Resources and Evaluation, pp.947–952, 2000.
[2] Takezawa, T., Nakamura, A. and Sumita E., “Database forConversational Speech Translation Research at ATR,” The
Phonetic Society of Japan, Vol.4, pp.16–23, 2000.
[3] Mima, H., Iida, H. and Furuse, O., “Simultaneous Interpretation Utilizing Example-based Incremental Transfer,”
Proc. of 17th Computational Linguistics and 36th Association for Computational Linguistics, pp.855–861, 1998.
[4] Watanabe, T., Okumura, A., Sakai, S., Yamabana, K., Doi,S. and Hanazawa, K., “An Automatic Interpretation System for Travel Conversation,” Proc. of 6th International
Conference on Spoken Language Processing, Vol. IV, pp.
44–68, 2000.
[5] Amtrup, J. “Incremental Speech Translation,” Lecture
Note in Artificial Intelligence, Vol. 1735, 1999.
[6] Kawaguchi, N., Matsubara, S., Takeda, K. and Itakura,F., “Multi-Dimensional Data Acquisition for Integrated
Acoustic Information Research,” Proc. of 3rd International Conference on Language Resources and Evaluation, pp. 2043–2046, 2002.
[7] Maekawa, K., “Corpus of Spontaneous Japanese: Its design and evaluation,” Proc of ISCA & IEEE Workshop on
Spontaneous Speech Processing and Recognition, 2003.
[8] Gile, D., “Consecutive vs. Simultaneous: which is moreaccurate?,” The Journal of the Japan Association for Interpretation Studies, No.1, pp. 8–20, 2001.