NEWSCAST SPEECH SUMMARIZATION VIA SENTENCE SHORTENING
BASED ON PROSODIC FEATURES
Kiyonori Ohtake1 , Kazuhide Yamamoto1,2 , Yuji Toma 3 , Shiro Sado3 , Shigeru Masuyama3 ,and Seiichi Nakagawa4
ATR Spoken Language Translation Research Laboratories, Kyoto 619-0288 Japan
Department of Electrical Engineering, Nagaoka University of Technology, Niigata 940-2188 Japan3
Department of Knowledge-based Information Engineering, Toyohashi University of Technology, Aichi 441-8580 Japan
4
Department of Information and Computer Sciences, Toyohashi University of Technology, Aichi 441-8580 Japan
E-mail: {otake,yamamoto}@fw.ipsj.or.jp, {nerusyu, masuyama}@smlab.tutkie.tut.ac.jp
12

ABSTRACTThis paper presents a speech summarizer that summarizes input
speech via several prosodic features, unlike models that use a speech
recognizer and conventional summarizing techniques proposed innatural language processing. Our approach analyzes the bordersof summary units by employing prosodic features of pitch, power,
and pause to summarize the input speech. Our summary generation trial implies robustness against noisy input compared withboth a sequential connection model of a speech recognizer and atext summarizer.
1. INTRODUCTION
Speech is an effective communication medium for human-to-humancommunication and is thus used commonly in daily life. As interest in the study of automatic document summarization in the fieldof natural language processing continues to increase, more attention is being paid to speech summarization, where both the input
and the output are speech, rather than text.
One may think that when we use a speech recognizer, the taskof speech summarization is the same as written text summarization[1].
While this may be so, there may also be an alternative approachto realizing a speech summarizer that processes speech directly.
Speech information involves not only linguistic information, butother types of information, such as emphasis and nuances. For example, conventional wisdom tells us that prosody must be somehow related, directly or indirectly, to syntactic structure. Thus,
there is room to try to build a summarizer that is suitable for speech,
using this information that text cannot effectively convey.
We present a new approach to summarize, or shorten, input
newscast speech by utilizing prosodic information. Throughout
the approach’s implementation, we do not use a speech recognizer
because we want to investigate the feasibility of a direct speechsummarizer that is not connected with a speech recognizer. One
advantage of this strategy is that the system is not affected by noiseand errors due to the recognizer, which is the biggest headache inspoken language processing.
In newscast programs, the sentences that are read by broadcasters tend to be very long. If we simply select some importantutterances as a summary of a news topic, the result of the summarization may not be acceptable because it may lack important
information. Therefore, we try to summarize each utterance byselecting some of its important parts.
–167–

We use prosodic information to determine the unit for summarization, to analyze the dependency structures[2, 3] between
the units, and to determine the importance of each unit. We willbuild a summarization method based on this idea, and apply it toJapanese broadcast news speech. In addition, we will compare theproposed method with a conventional method, i.e., using a speech
recognizer and a text summarizer to investigate the effect of directspeech summarization.
2. NEWSCAST SPEECH SUMMARIZATION
An overview of the summarization method is described in the following five steps. Step 1: Extract pitch pattern with the method
proposed by Hirose et al.[4]; Step 2: Extract accent phrases; Step
3: Detect summary units; Step 4: Analyze dependency structuresof summary units; and Step 5: Select important summary units.
We describe the details contained in and after Step 2 in thefollowing sub-sections.
2.1. Extracting accent phrases
An accent phrase, which is sometimes called a ‘prosodic phrase,’
can usually be observed as a fall-rise pattern of F 0, and the F 0contour of a sentence uttered continuously is considered to be aconnected series of these prosodic patterns. Based on this idea, we
extract phrase boundaries by detecting each rise in the patterns of
the F 0 contour.
To pick up the rise in F 0, we calculate the following formula:diff (t) =

1
w2

Z w/2

τ f (t + τ )dτ.

(1)

−w/2

Here, w represents the window size, and f (t) is F 0 at time t.
However, some t’s do not have an F 0 value, because F 0 is notobserved at unvoiced sounds. Although such times can usually beinterpolated, we use the last value of F 0 for an f (t) that has novalue. Moreover, this operation has little impact on the processes
that use this diff (t) value.
The beginning of an accent phrase is picked up as follows:
In an interval, where diff (t) > 0, the smallest t that maximizes
diff (t) is used, then the beginning of the accent phrase is given
as t0 = t − w/2. The end of one accent phrase is given as the
beginning of the next accent phrase.

2.2. Detecting summary units
A summary unit consists of consecutive accent phrases. We extract
unit boundaries by calculating regression lines that correspond to
fall-patterns of the base-pitch (Bp) contour. The base-pitch (Bp)
is a representative value of an accent phrase A, and is given by thefollowing formula:
Bp(A) = 0.2F 0max (A) + 0.8F 0min (A),(2)

where F 0max (A) indicates the maximum value of F 0 in phrase
A, and F 0min (A) indicates the minimum value of F 0 in phrase
A. We employ Bp(A) instead of F 0min (A) as the representativevalue of an accent phrase A to reduce some errors that are causedby the pitch extractor and other miscellaneous sources.
To approach the Bp contour with a regression line, we employ
a mel scale, which is defined by the following formula:m(f ) = 2410 log 10 (1 +

1.6f
),
1000

(3)

where f is the frequency in Hz.
We calculate regression lines lij (t) that minimize the meansquared error ij for all combinations of any two accent phrases(Ai , Aj , i < j).
lij (t) = aij t + bij
ij =1j −i+1jX(4)z(Ak )(m(Bp(Ak )) − lij (x(Ak )))2 , (5)k=iwhere, z(A) indicates the interval length that is not silent in phrase
A, and x(A) indicates the center time point of phrase A.
We preserve the regression lines that fulfill the two conditions,ij ≤ θ , aij < 0, and eliminate the remaining regression lines.
Regression lines that fulfill the following two conditions arealso eliminated.
• aij < −θs• In the accent phrases (Ai , Ai+1 , · · · , Aj ) corresponding tolij , there is a silent pause between any two adjoining phrases,
given as (Ak , Ak+1 , i ≤ k < j).
In addition, the regression line lij is eliminated, if li0 j 0 exists,
where i0 ≤ i, j ≤ j 0 .
The remaining regression lines basically correspond to summary units. Moreover, each accent phrase that does not correspondto a regression line is considered as a summary unit.
We introduce heuristics to adjust the boundaries of the summary unit, because there are mis-extractions of boundaries causedby pitch that rises to express emphasis or paraphrasing. We connect two summary units that sandwich such a boundary and unite
them as one unit.
The processing of connection is as follows:
If there is a silent pause whose length is less than or equalto θp between two consecutive summary units (Ck , Cl ), and the
beginning accent phrase Ai of Cl satisfies the following formula,then Ck and Cl will be connected.
Bp(Ai−1 ) ≤ Bp(Ai ) ≤ U p(Ai−1 ),(6)where the function U p(Ai ) expresses the upper 20% of the distribution of F 0 in the accent phrase Ai , as defined by the following
formula:U p(Ai ) = 0.8F 0max (Ai ) + 0.2F 0min (Ai ).

(7)
–168–

On the other hand, if there is a silent pause whose length ismore than θp between two consecutive summary units (Ck , Cl ),and the beginning accent phrase Ai of Cl satisfies the following
formula, then Ck and Cl will be also connected.
U p(Ai ) − Bp(Ai ) < U p(Ai−1 ) − Bp(Ai−1 )(8)2.3. Analyzing dependency structure
In this paper, we also introduce an easy dependency analysis method.
This method analyzes the dependency structures among summary
units of the input speech.
If there is a silent pause whose length is less than or equal to θpin any two consecutive summary units (Ck , Cl ), then Ck dependson (modifies) Cl . Otherwise, the summary unit depends on the lastsummary unit in the utterance.
2.4. Selecting summary units
The method for generating the summary is as follows: (1) Selectthe summary units, then (2) concatenate the selected units.
We introduced three selection methods. Method 1 is based oncommon heuristics for broadcast news texts, method 2 is based on
the heuristics of prosodic features derived from the observation ofmanually generated summaries, and method 3 is similar to method
2, but we free it from parameters.
The details of each method are as follows:[method 1](1) The first sentence: select all units. (2) The second sentence: eliminate the first summary unit. (3) After the third sentence: select the last two summary units for each sentence.
[method 2](1) Select all summary units in the first sentence. For all sentences after the second sentence, the following items are applied:(2) Eliminate summary units that hold the mean value of Bp for all
accent phrases in units more than or equal to a speaker-dependentthreshold, e.g., 150 Hz. (3) Eliminate summary units that have amean power less than or equal to 0.01 dB. (4) Eliminate summary
units that depend on other units eliminated by the above items.
The values 150 Hz and 0.01 dB, were determined by a preliminary parameter estimation experiment.
[method 3](1) Select all summary units in the first sentence. For all sentences after the second sentence, following items are applied: (2)Eliminate a summary unit if the mean value of Bp in the summary
unit is greater than or equal to the top 30% of the range of themean value of Bp in the sentence. (3) Eliminate summary units
that depend on other units eliminated by the above items.
3. EXPERIMENTS
We carried out several experiments to investigate the following
points: (a) Whether the summary unit is an effective unit in speechsummarizing, and (b) whether the speech summarized by the proposed method is natural.
We employed an NHK news speech database (16 kHz, 16 bit)for these experiments.
3.1. Estimating parameters
The summarizing method requires some parameters, which we determined in a preliminary parameter estimation experiment. ThreeTable 1. Detecting summary units with/without heuristics
(a)
(b)
(c)
P
R
F
with
123
87
112 70.7% 77.7% 74.0%
without 213 102 112 47.9% 91.1% 62.8%(a): # of detected boundaries P: Precision(b): # of hit boundaries
R: Recall(c): # of correct boundaries
F: F-measure

Table 3. Results for preservation ratio of important information
Avg Max Min
SD
Original
4.9
5.0
4.5
0.17
Method 1
3.9
4.5
2.5
0.69
Method 2
4.3
4.8
3.0
0.60
3.7
5.0
2.6
0.69
Method 3
Table 4. Results for phonological naturalness
Avg Max Min
SD
Original
4.9
5.0
4.3
0.24
4.0
4.5
3.5
0.50
Method 1
Method 2
4.4
4.8
3.9
0.33
3.6
4.3
2.4
0.58
Method 3

Table 2. Speech summarization ratio (SS-ratio) by three methods
Avg
Max
Min
SD
Method 1 72.4% 98.7% 53.2% 8.25%
Method 2 70.1% 96.1% 53.3% 7.55%
Method 3 80.7% 91.2% 74.7% 6.63%
articles, containing 15 sentences, were randomly selected from the
NHK news speech database. These articles were all uttered by amale broadcaster.
First, we estimated the following parameters by trial and error.
The estimated parameters were as follows: θe = 1000 mel2 , θs =0.1 mel/s, θp = 0.5, and w = 500 ms.
Second, we estimated some parameters for the selection method.
To estimate these parameters, we had eight subjects summarize
three transcribed articles by selecting the most important parts.
We created good examples of summaries by selecting parts chosen by more than three subjects. We then estimated the parameters
required for selecting methods based on these good examples.
3.2. Detecting summary units
Ten new articles, containing 52 sentences, were randomly selectedfrom the NHK news speech database. These articles were all uttered by a male broadcaster. We also prepared the correct summary unit boundaries for these ten articles by hand. We applied
our method to these articles and evaluated them by comparing theresults with the manually prepared answers. In addition, to evaluate the effectiveness of the heuristics introduced in Section 2.2, weapplied our method without the heuristics. The evaluation results
are shown in Table 1.
3.3. Evaluating speech summarization
We summarized ten new articles randomly selected from the NHKnews speech database with the three selection methods introducedin Section 2.4.
First, we calculated the speech summarization ratio (SS-ratio),which is defined by the following formula for each article.
SS-ratio =playback time of summary
× 100(%)
playback time of original(9)

Table 2 shows the results of the calculation for the average, maximum value, minimum value, and standard deviation (SD).
Second, we conducted a questionnaire-based survey with eightsubjects on the preservation ratio of important information andphonological naturalness of the summarized speech, and the original speech for the ten articles. The subjects evaluated each article
using a scale of 1 to 5 (best: 5, worst: 1) with respect to the twopoints above. We took the average of eight evaluations for each article, and computed the average, maximum value, minimum value,and standard deviation (SD) for all ten articles. Table 3 presents–169–

the results for the preservation ratio of important information, andTable 4 shows the results for phonological naturalness.
We also compared our method with a sequential connection
model of a speech recognizer and a text summarizer. This experiment was carried out as follows: Ten articles randomly selected from the speech database were transcribed by a speech recognizer. We then applied a text summarization method developedby Mikami et al.[5] to the transcribed article. Next, we summarized the article with our speech summarizer using selection
method 2, and mapped the result onto the transcribed article. Finally, we compared the two summarization results.
To compare the results, we had six subjects summarize the ten
articles by selecting the important parts, with the condition that the
text-based summarization ratio is 70%. We can create good examples of summaries by selecting the parts chosen by more than four
subjects; the average summarization ratio of these good examples
is 56.8%.
For those good examples, we calculated recall and precisionof two summaries. The results are shown in Table 5 with the summarization ratio (S-ratio).
4. DISCUSSIONS
4.1. Analysis of summary unit boundaries
In our approach we have introduced a heuristic, wherein base pitch
increases caused by a paraphrase or an emphasis expression are
not regarded as boundaries. This heuristic results in lower recalls,
and thus changes of summarization ratios, by changes of articles.
Eight instances related to this reason are observed among the 25
instances in which summary units were not recognized.
Some pauses tend to be inserted when parallel clauses are segmented in utterance. These pauses are mis-recognized by our approach as boundaries of summary units. Precision decreases down
for this reason, and it mainly affects phonemic naturalness. There
were 12 incidents observed among 36 analysis errors.
The parameters for summary unit boundary detection, fixedby the preliminary experiments, depended on each feature of thearticle in general. There were certainly some errors due to inapTable 5. Comparing two summaries
S-ratio SD of S-ratio
P
Ours
73.7%
12.7%
69.1%
Mikami’s 51.0%
20.8%
66.8%

R
48.3%
36.5%

propriate parameter values. This static parameter determination
causes both summarization ratio changes and errors in selection ofsummary units.
Some analysis errors due to the pitch extractor were also observed. For instance, voiceless sounds, such as a fricative /s/, areregarded as being spoken with no vibration of the vocal cords. It
therefore does not have a pitch and no detection is possible by apitch extractor. Consequently, in a part where no pitch information is observed, a distinction between fricatives and silent parts isrequired.
The dependency analyzer also made some errors: it judgedthat the dependent part was the predicate even when it was in fact
following part. This type of errors causes the analyzer to fail indeletion. Phonemic naturalness is affected by these errors.
The errors listed here were the major errors observed. Although many errors were observed in this experiment, the evaluations shown in Tables 3 and 4 illustrate a fine level of performance.
4.2. Comparison of Strategies
Tables 3 and 4 show that method 2 is the best method among thethree, considering both the preservation ratio of important information and phonemic naturalness, indicating that the thresholds
we have determined fit the inputs for the experiment. However, it
is known that the base pitch greatly differs between male and female newscasters. Moreover, the degree of power depends on thearticle, even when the speaker is the same. These two factors may
cause numerous performance changes.
The performance of method 1 was inferior to that of method
2, but better than that of method 3 because method 1 is less likely
to reduce the amount of important information or break phonemic
naturalness than the others, since it employs features that dependon newscast speech. However, it does not outperform method 2
because the inputs did not necessarily contain the features we had
expected.
As the maximum preservation ratio of important information
illustrates, method 3 works very well for some articles. However,
the method is the worst among the three. This result indicates thatthe method, in which the threshold for selecting summary units is
relatively determined by the pitch range of the input, did not work
for most of the articles.
4.3. Summarized Speech
We have obtained fine evaluations of summarized speech, for bothnaturalness of phonemes and in the preservation ratio of important
information. We assume that these evaluations originate in thecharacter of news speech, which is clearer and more well-formed
than ordinary daily speech. However, some speech in the NHKnews speech database included background sounds such as music,
which may decrease the performance. Thus, it can be said that ourapproach works well, even if there is some background noise in the
input. We expect that the proposed approach may be applicable not
only to broadcast speech, but also to more difficult tasks, such asmonologues spoken in lectures and meetings.
Moreover, in a situation where a speech recognizer may makemistakes in recognizing particles, better performance is expected
when using the parsing approach proposed in Section 2, as compared with that by Mikami et al.[5]. Unfortunately, Table 5 does
not show us the striking difference between the performances, measured by recall and precision, of the two approaches.
–170–

However, Table 5 does tell us that the result for the sequential
connection model was very unstable. Due to mistakes made by the
speech recognizer, it is difficult to determine the correct parts thatshould be deleted. The summarizer tends to delete long parts because it is difficult to determine the correct boundaries suited forsummary. On the other hand, our method achieved stable performance, and the summarization results were more natural.
5. CONCLUSION
This paper describes our attempt to build a speech summarizer
that summarizes input speech via several prosodic features, unlike
models that use a speech recognizer and the conventional summarizing techniques proposed in natural language processing.
Our experiments have proved that a summarizing strategy basedon the base pitch and power of the target summary units attains thebest performance among the three strategies that we compared.
However, to summarize a variety of speeches, we need to determine the parameters dynamically, and that depends on the characteristics of the target input.
Our summary generation trial has proved some advantages ofthe proposed method in terms of robustness against noisy input,
compared with a sequential connection model of a speech recognizer and a text summarizer.
Further investigations are necessary to (1) determine parameters, (2) recognize parallel structure against clauses, and (3) consider other prosodic features, such as speed.
6. ACKNOWLEDGMENT
This research was supported in part by the Telecommunications
Advancement Organization of Japan, Grants-in-Aid and the 21stCentury COE Program from the Ministry of Education, Science
and Culture of Japan. The authors would like to thank NHK (NIPPON HOSO KYOKAI) for providing Toyohashi University of Technology with the NHK news speech database.
7. REFERENCES[1] Chiori Hori, Sadaoki Furui, Rob Malkin, Hua Yu, and Alex
Waibel, “Automatic speech summarization applied to English
broadcast news speech,” in Proceedings of ICASSP 2002,
2002, vol. I, pp. 9–12.
[2] Yasuo Horiuchi and Akira Ichikawa, “Prosodic structure in
Japanese spontaneous speech,” in Proceedings of ICSLP 98,
1998, pp. 591–594.
[3] Norihiro Eguchi and Kazuhiko Ozeki, “Dependency analysis
of Japanese sentences using prosodic information,” The Journal of The Acoustical Society of Japan, vol. 52, no. 12, pp.
973–978, 1996, (in Japanese).
[4] Keikichi Hirose, Hiroya Fujisaki, and Shigenobu Seto, “A
scheme for pitch extraction of speech using autocorrelation
function with frame length proportional to the time lag,” inProceedings of ICASSP-92, 1992, vol. I, pp. 149–152.
[5] Makoto Mikami, Yuko Ishizako, Hirotaka Akamatsu, Shigeru
Masuyama, and Seiichi Nakagawa, “An experiment on generating captions by summarizing using speech recognition results of broadcast news,” in Proceedings of the 58th Meeting of
the Information Processing Society of Japan, 1999, vol. 3W-5,
pp. 275–276, (in Japanese).