Interpreting Unit Segmentation of Conversational Speechin Simultaneous Interpretation Corpus
Zhe DING*, Koichiro RYU*, Shigeki MATSUBARA**, Masatoshi YOSHIKAWA**Department of Information Engineering, Nagoya University**Information Technology Center, Nagoya University
Furo-cho, Chikusa-ku, Nagoya, 464-8601, Japan

ding@dl.itc.nagoya-u.ac.jp

Abstract
The speech-to-speech translation system is
becoming an important research topic with theprogress of the speech and language
processing technology. Considering efficiency
and the smoothness of the cross-lingual
conversation, the simultaneity of thetranslation processing has a great influence on
the performance of the system. This paper
describes interpreting unit segmentation of
conversational
bilingualspeechinsimultaneous interpretation corpus which hasbeen developed in Nagoya University. By
finding the segmentation point of spoken
utterances in the speech corpus manually, weidentified a clause-unit as a practicalinterpreting unit. In this paper, we examined
the availability of such unit, and segmented
spoken dialogue sentences into interpreting
units. A large-scale bilingual corpus for which
the interpreting units are provided can be used
for the simultaneous machine interpretation.

1 Introduction
In these years, with the progress ofinternationalization,naturalandsmoothcommunications on contact with computers incross-language conversation has been desired.
Therefore, the advance of technologies for speechprocessing and language translation has beenhighly expected, and the speech-to-speechtranslation system is becoming one of the mostimportant research topics.
Over the past few years, a considerable numberof studies have targeted the conversational speech,
and most of them are limited to the estimation ofdegree of accuracy. But nowadays, considering
efficiency and the smoothness of thecross-language conversation, the simultaneity ofthe translation processing attracts the attention of
all many researchers.
As to simultaneous machine interpretation, notonly the accuracy of the interpretation but its
output timing is also important, although theproper output timing is not well-defined. When asentence is recognized as an interpreting unit whichis said to be a linguistic chunk that could beinterpreted separately and simultaneously, thesimultaneity will not be satisfied. On the other hand,a small linguistic unit like a word or a phrase, etc. isnot an effective interpreting unit either, because it is
not necessarily realistic in current technologies ofspeech recognition (Ryu, 2004). Therefore, in thispaper we focused attention on a clause-unit as aninterpreting unit.
In this paper, we describe interpreting unit
segmentation of conversational bilingual speech insimultaneous interpretation corpus. The effectiveinterpreting unit is identified by finding thesegmentation of spoken utterances in bilingual
speech corpus. Added to this, we made aninvestigation into a possibility of simultaneousmachine interpretation by extracting suchinterpreting unit from our bilingual corpus(Tohyama, 2004). A large-scale bilingual corpus forwhich the interpreting unit is provided can be used
for the simultaneous machine interpretation.
This paper is organized as follows: Section 2
explains the concept of the interpreting unit
segmentation. Section 3 describes the preliminaryinvestigations. Section 4 describes the technique forannotating the bilingual corpus by the interpretingunits. Section 5 provides the result of an experimentand our observations of interpreting unit
segmentation.

2 Simultaneous Interpreting Unit
The conversational speech data of thesimultaneous interpretation corpus has beendeveloped in Nagoya University (Ryu, 2003). The
data consists of the conversational speech between
Japanese and English through the simultaneousinterpreters in traveling abroad situations such as
airport check-in or, booking of a room at a hotel.
The speech data of about 60,000 utterances and
420,000 words have been collected. This large-scalebilingual corpus provides the transcribed text
between Japanese and English, the bilingualadnominal
clause4%
if-clause“tara”
5%the others
6%

parallel-clause“de”
5%rationaleclause “node”
5%parallel-clause“ga”
5%emotional
phrase27%

subject “ha”
13%

quotationalclause6%“te”-clause
6%

Figure 1: A sample of the transcripts(2.1) ホテルの予約をして来なかったので/こちらでホテルを紹介していただきたいんですけれども(2.2) I haven’t made any hotel reservation /socould you introduce me any nice hotel?
This is an example of bilingual conversational
speech with interpreting units. Both Japanese andEnglish consist of two clauses and they aresemantically compliant each other. Therefore, we
can recognize each clause of Japanese asinterpreting units. When “ホテルの予約をして来なかったので” was input, the parallel interpreting“I haven’t made any hotel reservation” will beoutput.

3. Preliminary Investigations
In order to identify interpreting units in Japanesecontinuous
clause11%

Figure 2: Breakdown of the labels

if-clause “tara”
parallel-clause “ga”

clause boundaries of Japanesealignment, the visualization of speaking time, etc.
Figure 1 shows a sample of the transcript.
The main difference between consecutive
interpretation and simultaneous interpretation
would be the beginning time of the interpretation.
In general, in order to reduce listener’s waiting
time, simultaneous interpreters break up theutterance into several meaningful segments, andtranslate them incrementally. We call suchsegment interpreting unit. In other words,interpreting unit can be defined as a linguistic
chunk that could be interpreted separately andsimultaneously.
Recently, a small unit like word-unit orphrase-unit, etc. has been used as a unit of thesimultaneous machine interpretation though it is
not efficient and effective adequately, because it is
not necessarily realistic in current technologies ofspeech recognition. Therefore, in this paper wewill focus attention on clause-unit as a practicalinterpreting unit (Kashioka, 2004). The
simultaneous interpreting corpus which issegmented into practical interpreting units will begetting valuable in the coming machineinterpretation research.

discourse
marker7%

discourse marker
adnominal clausesubject “ha”
parallel-clause “de”
rationale-clause “node”“te”-clausecontinuous clausequotational clauseemotional phrase

0.0%

20.0%
40.0%
60.0%
80.0%
segmentation possibility

100.0%

Figure 3: Segmentation possibilities
conversational sentences, we made some provision
manually. We used the Japanese-to-English part inconversational speech data of the simultaneousinterpretation corpus, which has developed in theCenter for Integrated Acoustic Information Research(CIAIR), Nagoya University. We selected 11
dialogues randomly from the corpus. The dialogue
data consists of 519 spoken Japanese sentences intotal.
At first, we segmented the Japanese sentences into
clauses by using a clause boundaries detection
program, CBAP (Maruyama, 2004). In the result,
207 sentences were divided into two or more clauses.
The clause labels in these sentences are investigated.
Figure 2 shows the breakdown of the labels. We cansee that the top 11 clause labels of high occurrence
rate take over 94% of the total.
Then, we investigated whether these 11 kinds ofJapanese clauses can be identified as interpreting
units or not. The investigation was done by
extracting the segmentation points which satisfy thefollowing two conditions:z We can recognize the English boundary unit

corresponding to the detected Japanese clausesemantically.
z The corresponding boundary units ofJapanese and English appear in the sameorder.
That is, if a Japanese sentence can be segmented
into the boundary units A and B, its translation
into C and D, furthermore, A and C, B and D canbe aligned, respectively, then the boundary
between A and B can become a segmentation point.
This means that the boundary units A and B can be
regarded as interpreting units.
Figure 3 shows the rate of segmentation points
in the clause boundaries in a label-by-label basis.
We can see that the difference between "te"-clauseand continuous clause is greater, and therefore, we
identify the top eight clauses of this figure:if-clauses "tara", "te"-clause, etc. as interpretingunits.
In the result of an examination using the closed
data, the accuracy and the recall ratio were 78.9%
and 86.7%, respectively, we confirmed ouridentification method to be effective.

4 Interpreing Unit Segmentation
This section describes a technique forsegmenting a spoken Japanese sentence into twoor more interpreting units. Figure 4 shows theflow of the interpreting unit segmentation using aJapanese-English conversational speech corpus.
The technique consists of three steps: sentencealignment, sentence analysis, and sentence
segmentation. Each step will be explained in detailbelow.

4.1 Data Arrangement
The first step arranges the bilingual data because
the original text in the corpus was not separated bysentences. We used DETAG program to break theoriginal text up into sentences and take off fillers
which exert a harmful influence on analyzing
efficient interpreting units. Every sentence is end
up with a punctuation mark.
4.2 Language Analysis
The second step analyzes both Japanese andEnglish sentences linguistically, respectively. In
the below, let us use the following pair of alignedsentences (4.1) and (4.2) as an example. This
example was extracted from the CIAIR
conversational speech corpus in fact.
(4.1) あと日本の流行をお知りになりたいのでしたら若者が多く集まるエリアがございます。
(4.2) And if you want to know about Japanesefashion, there is an area which is crowded withyoung people.

Figure 4: The flow of the interpreting unit
segmentation
First, for the Japanese sentence, clause boundaries
are provided by CBAP to line up the candidates ofinterpreting unit segmentations. For example, (4.3)
is generated by applying the CBAP to (4.1).
(4.3) あと日本の流行をお知りになりたいのでしたら /if-clause"tara"/
若者が多く集まる /adnominal clause/
エリアがございます /sentence end/
Here, the labels of clause boundary units are wedged
between two slash symbols.
The result (4.3) indicates that the sentence (4.1) isdivided into three clause boundary units and theabove labels are provided for them. Among thelabels, both "if-clause" and "adnominal clause" areincluded in so called eight clause labels, which aredefined in the previous section. Therefore, threeclause boundary units are all the candidates ofinterpreting units.
On the other hand, for the English sentences,phrase structures are provided by RASP (Briscoe,2002), which is one of the context-free parsing
program, to define the syntactic fragments of thesentence. Since the RASP parser gives an English
sentence to a binary tree, the result is useful forfinding the corresponding segmentation points in a
top-down fashion. Figure 5 shows the parsing resultfor the English sentence (4.2).

4.3 Segmentation Into Interpreting Units
The last step extracts the interpreting units ofJapanese spoken sentences by considering theword-correspondence between the Japanese andEnglish sentences.
At first, the keywords in the sentences extractedusing the word-corresponding data. As a keyword,
the word whose part-of-speech are any one of noun,
adjective, and adverb, was extracted. The POS
tagging for Japanese sentences and English
sentences are executed by Chasen (Matsumoto,1999) and Brill's tagger, respectively. The resultfor (4.3) is (4.4), and for (4.2) is (4.5).
(4.4) あと(NN_1 日本)の(NN_2 流行)をお知りになりたいのでしたら /if-clause"tara"/ (NN_3
若者)が多く集まる/adnominal clause/(NN_4 エリア)がございます/sentence end/(4.5) And if you want to know about (NN_1Japanese) (NN_2 fashion), there is an (NN_4area) which is crowded with (NN_3 youngpeople)
Here, keywords are expressed as the bracketed
word with part-of-speech, the numbers shows theword correspondence.
Next, the keyword sequence are generated andthe segmentation points are extracted. For example,
the keyword sequences of (4.4) and (4.5) are as
follows:(4.6) (NN_1 日本) (NN_2 流行) /if-clause "tara"/(NN_3 若者) /adnominal clause/(NN_4 エリア) /sentence end/(4.7) (NN_1 Japanese) (NN_2 fashion) (NN_4 area)(NN_3 young people)
By considering the appearance order of thekeywords between Japanese and English, theboundary between the 1st and 2nd clauses in theJapanese sentence is extracted as interpreting unit
segmentation.
Finally, the segmentation points are provided for
the English sentence. It is required to find thesegmentation points in the sentence since those in
the keyword sequence are already decided. We
utilize the result of phrase structure parsing for that.
For example, there exists a segmentation point
between (NN_2 fashion) and (NN_3 area) in (4.7).
This means that any one of four word
segmentations in "(NN_2 fashion), there is an(NN_4 area)" is the segmentation point. It can beextracted based on the fragment segmentation inthe binary tree of Figure 5 because this tree showsthat this sentence (S) can be divided into "And if
you want to know about Japanese fashion" as aS
PP
CJ

S
NP

PP
P

VP
BES

NP

thereandNP

DETVP
VN

V

youwantSaTOtoNisifWHNP

V
V

VP

areaBEN

PPART

whichknowadv

APN

PPisPAfashionaboutVJapaneseNPcrowdedAPN

youngpeoplewithFigure 5 : Binary tree by RASP
prepositional phrase (PP) and "there is an area whichis crowded with young people" as a sentence (S).

5 Segmentation Experiment
In order to evaluate the effectiveness ofinterpreting unit segmentation of conversational
sentences and the feasibility of the technique which
has been explained in the previous section, we havemade a segmentation experiment. An experimental
data, we used the Japanese-to-English part inconversational speech data of the simultaneousinterpretation corpus. The data has 216 spoken
dialogues and 8721 sentences.
First, we tried to segment these sentences. Thereexisted 5019 clause labels with the exception of"sentence end". The total of the clause labels whichwere matching the top eight were 3846, and thesentences including at least one clause label in the
top eight is 2375. After applying the method of Step
3 described in section 4, we found 1005 clauselabels which can be recognized as interpreting unit
candidates, and 677 sentences which are including
such interpreting unit segmentation.
After examining the 1005 clause labels further,
we found there are some characters of them. Figure
6 shows the relation between the amount of thesentences with interpreting unit segmentation andthe amount of the interpreting segmentation in suchsentences. We may, therefore, reasonably conclude
that there are not a few sentences should besegmented even in conversational speech.
Figure 7 shows the rate of segmentation
possibility of the eight clause labels by using themethod of section 4 automatically. Comparing
Figure 4 with Figure 7, we may conclude that thesegmentation possibility of the eight clause labels
acquired by hand is differing greatly from the result
acquired by machine. From Figure 7, we can also
see that specific clause such as "discourse marker" isthe most difficult clause label to extract. The reason
may be thought as the amount of the keywords

amount of the sentence withinterpreting unit segmentation

500
450
400
350
300
250
200
150
100
50
0

important role for supporting natural and smoothcross-lingualmachine-mediatedspeechcommunication.

461

7 Acknowledgements

145
47
12

3

16 3 3 1 0 1
4 5 6 7 8 9amount of the interpreting unit segmentationin one sentence

Figure 6: Relation between sentences andinterpreting unit segmentation

segmentation possibility
0%20%

40%60%

80%100%

if-clause "tara"

Clause labels

parallel-clause "ga"discourse marker
adnominal clausesubject "ha"parallel-clause "de"rationale-clause "node""te"-clause
total

Figure 7: Rate of segmentation possibility of theeight clause labels
whichcanbealignedfrom
theword-correspondence data is not enough. For
example, if verbs can be extracted as keywords,
more practical interpreting unit may be extracted.

6Concluding Remarks

This paper has described a method forinterpreting unit segmentation of conversational
speech in CIAIR simultaneous interpretation
corpus. The segmentation is executed by extracting
specific clause boundaries in Japanese sentences
and by finding the segmentation points in thecorresponding English sentences based on wordalignment. We have made a segmentation
experiment using the conversational bilingualspeech. The result shows the possibility that thetop eight Japanese clause labels can be identified
as interpreting units. That is, when these clauselabels appear at Japanese speech, a simultaneousmachine interpreting system can break up thespoken sentences into two or more segments andtranslate them incrementally. The practicalinterpreting unit segmentation would play anThe authors would like to thank their colleague
Mr. Kazuya Tanaka for his valuable contribution inimplementation. They also wish to express their
gratitude to Dr. Hideki Kashioka and Dr. TakehikoMaruyama for their helpful suggestions. This
research was partially supported by the Grant-in-Aidfor Young Scientists (No. 17700148) of JSPS.

References
K. Ryu, S. Matsubara, N. Kawaguchi, and Y.
Inagaki, 2003. "Bilingual Speech Dialogue
Corpus for Simultaneous Machine Interpretation
Research",
ProceedingsofOrientalCOCOSDA-2003, pp. 217-224.
H. Tohyama, S. Matsubara, K. Ryu, N. Kawaguchi,and Y. Inagaki, 2004. "CIAIR SimultaneousInterpretation Corpus", Proceedings of Oriental
COCOSDA-2004, Vol. II, pp. 72-77.
T. Kashioka and T. Maruyama, 2004. "Segmentationof semantic unit in Japanese monologue", Proc. ofO-COCOSDA-2004, pp. 87-92.
T. Maruyama, T. Kashioka and H. Tanaka, 2004.
"Development and evaluation of Japanese clauseboundaries annotation program", Journal of
Natural Language Processing, 11(3):39-68,
2004.(In Japanese)
E. Briscoe and J. Carroll, 2002. "Robust accuratestatisticalannotation of general text", Proc. ofthe 3rd International Conference on Language
Resources and Evaluation, pp.1499-1504.
K. Ryu, A. Mizuno, S. Matsubara and Y. Inagaki,
2004. "Incremental Japanese Spoken Language
Generationin
Simultaneous
MachineInterpretation", Proc. of Asian Symposium on
Natural Language Processing to Overcome
language Barriers, pp. 91-95.
Y. Matsumoto, A. Kitauchi, T. Yamashita and Y.
Hirano, 1999. "Japanese Morphological Analysis
System ChaSen version 2.0 Manual", NAISTTechnical Report, NAIST-IS-TR99009.