Text Editing forLecture Speech Archiving on the Web
Masashi Ito1 , Tomohiro Ohno2 and Shigeki Matsubara3
1Graduate School of Information Science, Nagoya University
masashi@el.itc.nagoya-u.ac.jp
2Graduate School of International Development, Nagoya University
3Information Technology Center, Nagoya University
Furo-cho, Chikusa-ku, Nagoya, 464-8601, Japan
Abstract. It is very significant in the knowledge society to accumulate
spoken documents on the web. However, because of the high redundancyof spontaneous speech, the transcribed text in itself is not readable onan Internet browser, and therefore not suitable as a web document. This
paper proposes a technique for converting spoken documents into web
documents for the purpose of building a speech archiving system. The
technique edits automatically transcribed texts and improves its readability on the browser. The readable text can be generated by applying
technology such as paraphrasing, segmentation and structuring to thetranscribed texts. An edit experiment using lecture data showed the feasibility of the technique. A prototype system of spoken document archiving was implemented to confirm its effectiveness.
Keywords: spoken language processing, digital archiving, web contents,
paraphrasing, sentence segmentation

1Introduction

The enormous amount of information on the web have been forming an infrastructure for the so-called knowledge society. Most of such the information
is expressed in text at the present time. Hereafter, it is expected that not only
texts but also audio, video or other mixed media will be utilized as the knowledge
resources. Above all, speech is a kind of language media, and produced daily. In
fact, the amount of spoken documents being produced is overwhelmingly morethan that of written documents. If the speech such as discourse, lecture, monologue, conversation and discussion are accumulated on the web, the amount of
sharable information in the knowledge society would get much larger.
Several styles for archiving the spoken documents on the web can be considered. Though a style of uploading speech data would be very simple and would
have high feasibility, its significance from a viewpoint of reusability is not solarge. If the transcribed text is also archived in addition to the speech data, itseffectiveness would be increased in terms of accessibility. On the other hand,because of high redundancy of spontaneous speech, the transcribed text in itself is not readable on an Internet browser, and therefore not suitable as a web
document.

Fig. 1. Examples of the paraphrase rules and their applications

This paper proposes a technique for converting spoken documents into web
documents for the purpose of building a speech archiving system. The technique edits automatically transcribed texts and improves its readability on thebrowser. The readable texts can be generated by applying language technologies
such as paraphrasing, segmentation and structuring to the transcribed texts. An
edit experiment was conducted by using lecture speech data, and the result has
shown the feasibility of the technique. Furthermore, a prototype system of spoken document archiving was implemented in order to confirm its effectiveness.
This paper is organized as follows: Section 2 discusses spoken document
archiving on the web. Section 3 describes an edit technique of spoken documents. Section 4 reports evaluation of the technique. Section 5 describes ourspeech archiving system.

2Spoken Document Archiving on the Web

We have aimed at constructing an environment where spoken documents arearchived as web documents. In general, a web document is described by themarkup language such as HTML, and created in consideration of the readability
on a browser. In that sense, a spoken document is not suitable for being readwith an Internet browser. Therefore, a transcribed text needs to be converted
into an HTML text.
As a research on speech archiving for Web contents creation, Bain et al.
have constructed a system named ViaScribe in order to improve accessibility ofspeech data [1]. ViaScribe targets lecture environments and creates a learningmaterial into which audio, video, slides and transcribed texts are integrated.
The learning material is uploaded to the Web, then the participants can accessit with a Internet browser. ViaScribe inserts linefeeds into the transcribed text at
places where pauses occur in order to improve the readability. Although Bain etal. target English speech, redundancy in spoken documents is larger in Japanesespeech. Therefore only the linefeed insertion cannot achieve enough readability.

Aiming at improving readability of a text based on the discourse structure,
Shibata et al. have proposed a method of automatically generating summary
slides from a text [2]. They have clearly specified the relations between sentences
or clauses by itemization. However, their research differ from ours in that theunits such as words or phrases are not targets. Furthermore, treating the written
language, they do not consider the features of spoken language.
As a technique for converting the transcribed text, there exists research on
automatic summarization [3, 4, 5, 6] and conversion from spoken language towritten language [7]. However, they are all conversions between plain texts. Incontrast, we propose a method for converting it into a web document, which canbe utilized as a fundamental technology in the archiving system.

3

Text Editing of Spoken Document

The conversion of spoken documents into web documents is executed by editing
spoken sentences so that they become easy to read. The conversion is realized by
performing paraphrasing of expressions, segmentation into short sentences andstructuring of a sentence in sequence.
3.1

Paraphrasing

In spoken language, there are a lot of expressions which are not proper as written
language expressions. Such expressions can be converted into readable expressions by paraphrasing them into other expressions.
Paraphrasing is classified into three types: deletion, replacement and insertion, according to the kind of target expressions. In deletion, redundant expressions particular to spoken language are removed. There exists not only fillers(uh, er etc.) and hesitations but also other redundant expressions.
(1) 正常化 ということ が行われた (the thing which is called normalization isperformed)“ということ (the thing which is called)” is redundant, and the readability isimproved by removing it. In replacement, expressions which are not proper as
written language are replaced with other expressions. For example, in the case of(2) 日本の本 なんか を読みました (I read Japanese books, etc.)“なんか (etc.)” is an expression particular to spoken language. By replacing “なんか (etc.)” with “など (etc.)”, it becomes a proper expression. Furthermore, inhonorific expressions, polite expressions, and so on, there are several cases wherereplacement is needed. In insertion, expressions which do not appear in spoken
language are added.
In order to realize the three types of paraphrasing mentioned above, wecreated paraphrasing rules by using the actual lecture speech data [8]. Fig. 1
shows examples of the rules and their applications. Here, in insertion, sinceconsidering the meaning is required, it is difficult to create the rules by only
surface information. Therefore, the target of this paper is limited to deletion
and replacement.

鈴木が何を言うかは ご察知とSuzukiwhatwill say

suzuki-ga

nani-wo

yu-ka-wa

go-sacchi-to

思いますがomoi-masu-ga

ご指名ですのでgo-shimeidesu-node

キックオフを するという 感じで 話しますkikkuofu-wosuru-toyukanji-dehanashi-masu

by theI will talkyou may
ThoughI’m adopted, so a kickoff
doingfeel ofseeI think thatThough I think that you may see what Suzuki will say, I’m adopted , so I will talk by the fell of doing a kick-off.

鈴木が何を言うかはご察知と思います。(I think that you may see what Suzuki will say.)
ご指名ですのでキックオフをするという感じで話します。 (I’m adopted, so I will talk by the fell of doing a kick-off.)
Fig. 2. Example of sentence segmentation

Fig. 3. Detection of the parallel phrases

3.2Segmentation

In sentence segmentation, one sentence is divided into two or more sentences. In
the case of a long sentence, the structure of the sentence is simplified and thus
it becomes easy to read. In segmentation, a sentence is divided at any of wordboundaries, and the final bunsetsu 4 of each divided sentence is transformed
into a proper expression as the sentence-end bunsetsu. Fig. 2 shows an exampleof sentence segmentation.
(3) 鈴木が何を言うかはご察知と思いますがご指名ですのでキックオフをするという感じで話します。(Though I think that you may see what Suzuki will say,
I’m adopted, so I will talk by the feel of doing a kick-off.)
In this example, by inserting a sentence boundary between “思いますが (ThoughI think that)” and “ご指名ですので (I’m adopted, so)” and transforming “思いますが” into “思います (I think that)”, the following sentence is generated:(4) 鈴木が何を言うかはご察知と思います。ご指名ですのでキックオフをするという感じで話します。 (I think you may see what Suzuki will say. I’m adopted,
so I will talk by the feel of doing a kick-off.)
The place into which a sentence boundary should be inserted is limited.
For analysis about the place, we divided sentences in the transcribed lecture
speech text by hand. About 89% of 299 sentence divisions are performed justafter a conjunctive particle “keredomo”, “ga”, “node”, “shi”, “to”, “kara”, orrenyou-tyuushi bunsetsu. Here, renyou-tyuushi bunsetsu is a bunsetsu of which
the morphological information is one of the following two cases: adverbial form
(e.g. “食べ,” “訳で”) and adverbial form + conjunctive particle “te” (e.g. “食べて,” “書いて”) .
4Bunsetsu is a linguistic unit in Japanese that roughly corresponds to a basic phrase
in English. A bunsetsu consists of one independent word and zero or more ancillary
words.

Since the above-mentioned conjunctive particles and renyou-tyushi bunsetsu
correspond to a predicate, they often become the ends of semantically meaningful units. In our research, these are considered to be the candidates of sentenceboundaries. However, a sentence can not necessarily be divided at all the candidates of division points. In some cases, the meaning of the original sentence may
be lost by dividing at the candidate. Therefore, we established the conditionof sentence segmentation based on dependency structure. That is, a sentence issegmented at a division point if no bunsetsu, except for a bunsetsu located justbefore the point, depends on some bunsetsu located after the point.
3.3Structuring

In our research, we structure a sentence to make it easy to read. Concretely
speaking, parallel structures are detected, and displayed by being itemized. The
parallel structures in a Japanese sentence can be detected based on the dependency structure. We use JUMAN [10] as morphological analysis and KNP [9] asdependency parsing.
Fig. 3 shows the detection of the parallel phrases in the following sentence:(5) 今我が国が行った行為や他国民に与えた苦痛を認識する．(Now, I recognize theactions taken by our country and the pains of other country peoples causedby them.)
In this sentence, noun phrase “我が国が行った行為や (the actions taken by ourcountry)” and “他国民に与えた苦痛を (the pains of other country peoples causedby them)” have a parallel relation with each other.
3.4

Example of sentence conversion

As an example of text edit by our technique, we describe the edit process of thefollowing sentence:(6) 西半球には米州機構というのがありますし欧州ヨーロッパには皆様ご承知のよ
うな EU とか全欧州安保協力機構というのがありますしアフリカにはアフリカ統一機構というのがあるわけです。(In the Western Hemisphere, the Organization of American States exists, and in Europe, the EU and Organization forSecurity and Cooperation in Europe exist, and in Africa, the Organization ofAfrican Unity exists.)
First, in paraphrasing processing, paraphrasing rules are applied to the underlined parts.
(7) 西半球には米州機構 というの がありますし欧州ヨーロッパには皆様ご承知のようなEU とか全欧州安保協力機構 というの がありますしアフリカにはアフリカ統一機構 というの が あるわけです。
Second, in segmentation processing, the sentence is divided into three sentencesby slashes right after the underlined bunsetsus.
(8) 西半球には米州機構が ありますし//欧州ヨーロッパには皆様ご承知の EU とか全欧州安保協力機構が ありますし//アフリカにはアフリカ統一機構があります。
Last, in structuring processing, our technique identifies the parallel phrases byでそれから今までのPKOがどちらかといえば軍隊中心のPKOであったとすればこれからはシビリアンですね警察官とか人権モニターとか人道援助関係者とかそういう行政の監視官とかそういう人達も加わったいわばミックス型のPKOになるであろうというふうに考えられます。西半球には米州機構というのがありますし欧州ヨーロッパには皆様ご承知のようなEUとか全欧州安保協力機構というのがありますしアフリカにはアフリカ統一機構というのがあるわけです。
And then, if the previous PKO is considered to have been rather the army-centered PKO, it is thought that the future PKO will become the civiliancentered PKO, as it were, the mixed type PKO in which people, such as police officers, monitors of human rights, humanitarian supporters, andgovernment observers, participate. In the Western Hemisphere, the Organization of American States exists, and in Europe, the EU and Organizationfor Security and Cooperation in Europe exist, and in Africa, the Organization of African Unity exists.

それから今までのPKOが軍隊中心のPKOでありこれからはシビリアン 警察官 人権モニター
 人道援助関係者
 行政の監視官
そういう人達も加わったミックス型のPKOになるだろうと考えられます。
西半球には米州機構があります。
ヨーロッパには
 EU
 全欧州安保協力機構があります。
アフリカにはアフリカ統一機構があります。

and then, if the previous PKO has been the army-centered PKO
civilians such aspolice officers
monitors of human rights
parties of humanitarian supports
government observers
includes such people
it is thought that it will be mixed type PKO
in the Western Hemisphere, Organization of American States exists
in EuropeEU
Organization for Security and Cooperation in Europe
exist
in Africa, Organization of African Unity exists

Fig. 4. Conversion of the transcribed text into the edited text

detecting bunsetsus which have parallel relation with each other, based on thedependency structure of the second sentence, and then, the parallel phrases are
itemized and displayed.
(9) 欧州ヨーロッパには皆様ご承知の EU とか全欧州安保協力機構が あります。
Fig. 4 shows the conversion of the transcribed text into the edited text. We cansee that it becomes easy to read by the conversion.

4Evaluation

To evaluate our method, we conducted an experiment on text conversion. As
the experimental data, we used the transcribed data of five Japanese lecturesin the simultaneous interpretation database [8]. We created the correct data oftext conversion by editing each sentence in the above-mentioned data by hand.
Evaluation is conducted based on each precision and recall for paraphrasing,segmentation and structuring. In addition, the precision is defined as the ratioof the correctly edited places to the places edited by our method, and the recall
is defined as the ratio of the correctly edited places to the manually edited placesin the correct data.
Fig. 5 shows an example of successful conversions. A total of 1452 places wereparaphrased. The precision and recall were 72.5% and 66.5%, respectively. As
the result of error analysis, we found out that redundant expressions were notfully deleted. This is because the paraphrasing rules for sentence-end expressions
which contain redundant expressions are not fully created.
Next, segmentation was performed at a total of 598 places. The precision andrecall of segmentation were 50.5% and 59.3%, respectively. Our segmentation
technique does not take into account the length of a sentence. Therefore, even if
an original sentence is short or each sentence generated by dividing the originalそれから三番目の時期は千九百八十九年のこのベルリンの壁の崩壊そしてドイツの再統一という大変歴史的な事件の起こった年でございますがそれ以降今日までというふうに私は分けております。
I defined the third period from 1989, in which the very historicalincidents such as collapse of the Berlin Wall and the reunification

それから三番目の時期は千九百八十九年のこの・ベルリンの壁の崩壊そして・ドイツの再統一という大変歴史的な事件の起こった年です。
それ以降今日までと私は分けています。

of Germany happened, to today.

Fig. 5. Example of a success of the sentence edit

lecturetitle
links toeach topic

video &speecheditedtext

鈴木，キーノート，キックオフ，キックオフ，出口さん，バイオリニスト，
keywords

search国連，国籍，日本語，公用語
日本，国際社会キーワード：
国際社会

国際社会[5] 日本としては戦争に負けても国土もすっかり荒廃して新たににもう一度受け入れてもらいたいということで努力をした時期でございます。

Fig. 6. Speech archiving system RISA

one becomes too short, our method segments the original one. In that case, thereadability is not necessarily improved by our segmentation.
Finally, structuring was performed at a total of 201 places. The precisionand recall were 32.0% and 25.9%, respectively, both were low. A possible reason
is the problem with the accuracy of morphological and dependency analysis.
Especially the dependency accuracy on a sentence including parallel structures
was not high, and then it materially affected the performance of structuring.

5Speech Archiving System

We implemented a speech archiving system named RISA according to our technique which has been explained in section 3. The system converts recorded spoken documents into web content for the purpose of their circulation on the Internet. The content consists of five components: text, sound, video, keyword, andsearch interface (as shown in Fig. 6). The texts are created by executing linefeed
insertion and topic segmentation after text editing. They are saved as HTMLfiles. We used the speech synthesis software HitVoice [11] for generating soundcontent, and TVML (TV program Making Language) [12], which is provided by
NHK Science & Technical Research Laboratories, for video content generation.
The users can directly access to the subcontents because text, sound and video
are aligned in a fine-grained fashion. The important words in the text are presented as keywords. The search component outputs the corresponding topic tothe query input. The set of keywords enables broad viewing of the content ofa spoken document. The links of each topic enable topic-by-topic browsing andare suitable for selective reuse.

6Conclusion

This paper has proposed a technique for converting speech transcripts into Web
texts for lecture contents archiving. The technique recognizes speech and edits
the transcribed text. Paraphrasing, segmentation, and structuring enable removal of unsuitable phrases and redundant expressions in written language. The
experiment using lecture speech has shown the effectiveness of the technique.
The speech archiving system was implemented to confirm its feasibility.
Acknowledgements This research was partially supported by the Grant-inAid for Scientific Research (B) of JSPS and by The Asahi Glass Foundation.

References
1. Bain, K., Basson, S., Faisman, A. and Kanevsky, D.: Accessibility, transcription,and access everywhere. IBM System Journal 44 (2005) no. 3 589–603
2. Shibata, T. and Kurohashi, S.: Automatic Slide Generation Based on Discourse
Structure Analysis. Proc. 2nd IJCNLP2005 (2005) 754–766
3. Chatain, P., Whittaker, E.W.D., Mmzinski, J.A. and Furui, S.: Topic and Stylistic
Adaptation for Speech Summarisation. Proc. IEEE ICASSP (2006)
4. James, C. and Mirella, L.: Models for Sentence Compression: a Comparison Across Domains, Training Requirements and Evaluation Measures. Proc.
ACL/COLING2006 (2006) 377–384
5. Zhu, X. and Penn, G.: Summarization of Spontaneous Conversations. Proc. 9thICSLP (2006) 1531–1534
6. Murray, G., Renals, S., Carletta, J. and Moore, J.: Incorporating Speaker andDiscourse Features into Speech Summarization. Proc. HLT (2006) 367–374
7. Shitaoka, K., Nanjo, H. and Kawahara, T.: Automatic Transformation of Lecture
Transcription into Document Style Using Statistical Framework. Proc. 8th ICSLP(2004) 2169–2172
8. Matsubara, S., Takagi, A., Kawaguchi, N. and Inagaki, Y.: Bilingual Spoken Monologue Corpus for Simultaneous Machine Interpretation Research. Proc. 3rd LREC
(2002) 153–159
9. http://nlp.kuee.kyoto-u.ac.jp/nl-resource/knp.html
10. http://nlp.kuee.kyoto-u.ac.jp/nl-resource/juman.html
11. http://www.hke.jp/products/voice/voice index.htm
12. http://www.nhk.or.jp/strl/tvml/english/player2/index.html
This article was processed using the LATEX macro package with LLNCS style