Conversation summarization using machine learning and scoring method
Kazutaka Shimada, Shinpei Toyodome and Tsutomu Endo
Department of Artificial Intelligence, Kyushu Institute of Technology
680-4 Iizuka Fukuoka 820-8502 Japan{shimada, s toyodome, endo}@pluto.ai.kyutech.ac.jp

Abstract—In this paper, we propose a method for conversation summarization. For the method, we combine two approaches; a scoring method and a machine learning technique.
We extract important utterances with high confidence froma conversation by using the scoring method. However, thenumber of extracted utterances is not enough as a summary.
To solve this problem, we incorporate utterances extracted
by SVMs to the summary. For the integration process, we
compare some approaches. In the experiment, our method
generated balanced summaries in terms of the summary lengthand readability, as compared with a method with SVMs only.

Dialogue

Scoring

Integration

Summary

Keywords-Conversation summarization, Scoring, SVMs, Integration.
Figure 1.

I. I NTRODUCTION
Multi-party conversation is a communication that involves
three or more participants with utterances. There are many
types of multi-party conversation such as spontaneous dialogues, meetings and chats on the Web. To understand thecontent of a conversation easily, the summarization has animportant role.
In this paper, we propose a method for conversation summarization. Traditional summarization studies have handled
a single document or multi-documents as the target [5].
Many studies in the summarization are based on extraction
approaches [2], [13]. In these approaches, the systems extract sentences on the basis of term frequency, location, cuewords and so on. Our method is also based on an extraction
approach.
The target data in this paper is multi-party conversation.
For conversation summarization, relations between utterances are more important, as compared with document
summarization such as news papers. Higashinaka et al.
[4] have proposed an improved HMM-based summarization
method for contact center dialogues. The dialogue in acontact center consists of utterances between two persons.
We handle conversations with four persons as the target
data. The data are free conversation about a topic. In otherwords, our multi-party conversations are more spontaneousand not well-structured. Therefore, relations between utterances are more complicated. Xie et al. [11] have evaluated the effectiveness of different types of features. They
compared lexical, structural, discourse and topic features
for machine learning. However, utterances in conversations
usually contain anaphoric relations. Lack of these relations

Machine learning

The outline of our method.

in a summary leads to decrease of readability. To solve thisproblem, we introduce features about anaphora.
For the conversation summarization, we combine two
approaches; a scoring method and a machine learning technique. We extract important utterances with high confidence
from a conversation by using the scoring method. However,the number of utterances extracted by the scoring method is
not enough as a summary. To solve this problem, we incorporate utterances extracted by SVMs to the summary. We
apply three integration approaches to summary generation.
In the experiment, we compare the proposed method with abaseline method based on SVMs only.
II. M ETHOD
In this section, we explain our proposed method.
A. Outline
Our method extracts utterances in a conversation on thebasis of the importance and relations between utterances inthe summarization process. We use two approaches for theprocess; a scoring method and a machine learning technique.
Figure 1 shows the outline of our method.
The purpose of the scoring process is to extract the mostimportant utterances in each conversation with high precision. We extract utterances exceeding a threshold. Although
the number of utterances is small, the extracted utterances
are the core of a summary.
The second method is a machine learning technique. We
use Support Vector Machines (SVM) [9]. The purpose of themethod based on SVMs is to extract important utterancesthat are not extracted by the scoring method. The output
of SVMs contributes to the improvement of the recall rate
about a generated summary.
In the integration process, we combine the outputs fromthe two approaches. In this paper, we propose three patterns
for the integration.

Then, we compute the score of each utterance by using
three scores on word-level, namely key1, key2 and key3.
∑
sent1(U ) =key1(w)(4)w⊂T

sent2(U ) =∑key2(w)(5)

key3(w)(6)w⊂T

sent3(U ) =∑w⊂T

B. Scoring
We compute a score of each utterance in a conversation.
We apply the panoramic view system proposed by Sunayama
and Yachida [8] to the scoring process. The method computed a score by word frequencies and conditional probabilities based on the co-occurrence frequency of words insentences. They defined three types of keywords, (1) basic
keywords, (2) topic keywords and (3) feature keywords, andthen computed a score of each sentence by using the scoresof the three keywords.
First, we divide each utterance in a conversation to morphemes by using a morphological analyzer1 . In this paper,
we handle nouns2 , verbs and adjectives in each utterance.
The score of a basic keyword is based on the frequency.
key1(w) =f requency(w)(1)

Next, we compute the second score. They defined words
which often appear together with the basic keywords as topic
keywords. The score of a topic keyword is computed asfollows:
key2(w)=∑ n(w ∩ g)
g⊂G

n(g)

=

∑ n(w ∩ s)
s⊂S

n(w)

(3)

where S is the set of topic keywords. The purpose ofthis score is to detect words that appear only in sentences
containing the topic keywords. It is based on the idea thatwords consistent with the flow of topic keywords forming themain topic of the text and not appearing in other sentences
are given high evaluations.
1 We used Mecab. http://mecab.sourceforge.net/
2 We use nouns of which the frequencies are more than 2.

score(U )=sent1(U ) + sent2(U ) + sent3(U ) (7)Here we add a new factor to the scoring. We assume thata long utterance has an important role in a conversation.
Therefore, we introduce a weighting factor basing on thenumber of morphemes in each utterance.
f inalScore(U ) =score(U ) ×

M orpT
AveM orp(8)

where M orpT is the number of morphemes in the utterance
U and AveM orp is the average number of morphemes in
one utterance in the conversation.
The purpose of the scoring process is to extract the mostimportant utterances in each conversation with high precision. Therefore we set a strong limitation on the extraction.
In this paper, we regard utterances with the top 10 % scoreof all as the important utterances in this scoring method3 .
C. Machine learning(2)

where G is the set of basic keywords and n(g) is the numberof utterances containing a basic keyword g. n(w ∩ g) is theco-occurrence of w and g. We use words in the top 10 %of all as topic keywords.
Finally, we detect feature keywords and compute thescore.
key3(w)

By using these equations, we obtain three scores onutterance-level. The score of an utterance is computed by
the summation of these scores.

One approach to extract important information is to utilize machine learning techniques. We apply SVMs to thissummarization process. We use 19 features for SVMs. They
are classified as (1) features in an utterance, (2) features
between utterances, (3) features about anaphora and (4)features based on scores.
1) Features in an utterance: The first feature category
consists of four features focusing on each utterance itself.
• Length:
Long utterances include much information and often
contain high potential as important utterances. Therefore, we use the number of morphemes as the feature.
• Presence of word:
We set TRUE to the feature if an utterance contains aword that occurs twice or more in a conversation.
• Presence of verbs and adjectives:
In spontaneous conversations, utterances without any
verbs and adjectives often exist. We apply the presencesof verbs and adjectives to the feature.
3 Here we use another limitation. Our method does not extract three ormore consecutive utterances by one person. This is a heuristic rule.

Interrogative:
Interrogative utterances often have an important rolein a conversation because they are a turning point of
a topic and contain strong context for previous andnext utterances. Therefore, we use the presence ofinterrogative as the feature.
2) Features between utterances: The second feature category consists of six features about relations between utterances.
• Difference of length:
The next utterance of an utterance with high importancesometimes consists of a small number of words, suchas expressions about agreement. Therefore we use thedifference of the utterance length between the current
utterance and the next three utterances4 .
• Presence of word in the previous utterance:
We also set TRUE to this feature if the previous
utterance contains a word that occurs twice or morein a conversation.
• Presence of interrogative in the previous utterance:
We also use the presence of interrogative in the previous
utterance as the feature.
• Same word in the previous utterance:
If a word in the current utterance is included in theprevious utterance, the current utterance contains highpotential as important utterances because it indicatesthat the two utterances contain strong context. We
handle nouns, verbs and adjectives for the feature.
• Same word in the next three utterances:
We also use the same word feature for the next threeutterances.
• Consecutive utterance:
If one person utters continuously, the second utterance
sometimes is supplemental information. It often boosts
the importance of the utterance. Therefore we use thepresence of consecutive utterances of one person as thefeature.
3) Features about anaphora: Anaphoric relation is one of
the most considerable points in conversation summarization
tasks. Nishikawa et al. [6] have reported the significanceof handling of anaphoric relations. If a summary contains
an utterance with an anaphora and does not contain an
utterance with the antecedent, the readability of the summary dramatically decreases. Therefore, the features aboutanaphoric relations are of extreme importance. The thirdfeature category consists of six features from three pairs.
• Referring expression in the current utterance and referring expression in the next three utterances:
These features are based on the presence of referring
expressions such as “kore (this)” and “sotti (there)”.
• Connective expression in the current utterance andconnective expression in the next three utterances:•

4 Actually, the difference is based on the number of morphemes.

These features are based on the presence of connective
expressions such as “demo (but)”, “shikamo (furthermore)” and “tadasi (unless)”.
• Response expression in the current utterance and response expression in the next three utterances:
These features are based on the presence of response
expressions such as “hee (heh)” and “un (Yes)”.
4) Features based on scores: The fourth feature category
consists of three features based on the scores in Section II-B.
• Score of basic keywords:
We use the score of sent1 (Eq. 4).
• Score of topic keywords:
We use the score of sent2 (Eq. 5).
• Score of feature keywords:
We use the score of sent3 (Eq. 6).
Here, these scores are normalized in 0 to 1.
D. Summary generation
The previous and next utterances of the important utterance extracted by our method are usually significant in theconversation summarization. They have an impact on thereadability of the generated summary.
In this paper, we apply three approaches to the summary
generation process. The basic summary consists of utterances by the scoring method including Eq. 8.
MethodP N :This method adds the previous and next utterancesof each utterance in the basic summary to thesummary.
MethodM L :This method adds utterances by SVMs to thesummary. It selects one utterance with the highoutput score of SVMs between utterances in thebasic summary.
MethodP N M L :This method is the combination of MethodP N and
MethodM L
Figure 2 shows an example of each approach.
III. DATASETFor the machine learning and evaluation, we need a tagged
corpus with an importance degree of each utterance. In thispaper, we use the conversation corpus used in [7]. It consistsof 10 spontaneous conversations with 1615 utterances5 . The
number of participants in each conversation is 4 persons. The
participants had a talk about “Movies”, “Games”, “SNS” andso on.
For the conversations, three annotators judged the importance degree of each utterance in a phased manner.
Figure 3 shows the process. We regard all utterances ineach conversation as level-1. First, the annotator selected
three quarters of utterances from all utterances (level-2).
5 All utterances in the conversations are in Japanese.

Original Dialogue

Extracted by Scoring(Basic Summary)
Utterance 2
Utterance 10

Utterance 1
Utterance 2
Utterance 3
Utterance 4
Utterance 5
Utterance 6
Utterance 7
Utterance 8
Utterance 9
Utterance 10
Utterance 11
Utterance 12
Utterance 13
Utterance 14
ࠉ㸬㸬
㸬㸬㸬
㸬

Our method
without Eq. 8

Precision
0.948
0.873

Recall
0.194
0.197

F
0.323
0.321

Ranking by SVMs
1. Utterance 5
2. Utterance 2
3. Utterance 6
4. Utterance 10
5. Utterance 1
6. Utterance 14
7. Utterance 9
ࠉ㸬㸬㸬
㸬㸬㸬

Method ML

Method PN

Utterance 2
Utterance 5
Utterance 10

Utterance 1
Utterance 2
Utterance 3
Utterance 9
Utterance 10
Utterance 11

Figure 2.

Table IR ESULT OF THE SCORING

Method PNML
Utterance 1
Utterance 2
Utterance 3
Utterance 5
Utterance 9
Utterance 10
Utterance 11

An example of the summary generation.

all utterances (level-1)
select3/4 of alllevel-2 utterances
select1/2 of alllevel-3 utterances

select1/4 of alllevel-4 utterances
Figure 3.

The annotation process.

Next, the annotator selected a half of utterances from theselected utterances (level-3). Then, the annotator selected aquarter of utterances from the level 3 utterances (level-4).
Finally, we selected utterances obtaining the average levelof three annotators which was more than 3, as the importantutterances for the summarization.
The agreement between annotators is as follows: 0.59 for
the Annotator 1 and 2 and 0.57 for the Annotator 1 and 3.
Both the κ values are approximately 0.3 and not high6 . We
also compute the agreement as a two-class problem. In otherwords, we integrate the level 3 and 4 to “important” and 1
6 The average value of the mean squared errors between them, namely
the annotator 1, 2 and 3, is approximately 1.2.

Table IIR ESULT OF SVM
Feature
(1)
(2)
(3)
(4)
ALL

Precision
0.789
0.754
0.539
0.728
0.800

Recall
0.682
0.626
0.650
0.561
0.740

F
0.732
0.684
0.589
0.634
0.769

and 2 to “not important”. In this situation, the agreements
are 0.75 and 0.73 and the κ values are 0.50 and 0.46.
IV. E XPERIMENTIn this experiment, first, we evaluated each utterance
extraction method; Scoring and SVMs. Then, we evaluatedthe readability of the generated summaries.
A. Accuracy of each method
First, we evaluated the scoring method. Table I shows theexperimental result. The scoring method with Eq. 8 outperformed the method without that in terms of the precision
rate. The purpose of the scoring method is to construct thebasic summary for the summarization process. Therefore,
the scoring method with Eq. 8 was suitable, as comparedwith the method without that.
Next, we evaluated our method based on SVMs with 10fold cross validation. We used the data mining tool WEKA
[3] for the implementation. In the experiment, we compared
the effectiveness of each feature category; (1) features in an
utterance, (2) features between utterances, (3) features aboutanaphora and (4) features based on scores. Table II showsthe experimental result. The most effective feature category
was (1) features in an utterance. In particular, the lengthfeature was effective for the accuracy. The best accuracy
was produced by the combination of all features.
B. Evaluation of summary
Next, we evaluated the outputs of the summarization
process. We generated summaries from 5 conversations ina qualitative manner. We compared our three methods witha baseline. The baseline summary consisted of all outputs
from SVMs. Figure 4 shows an example of the output of
our method. In the figure, the utterances with a rectangle
are the output from the scoring method. The utterances with“**” are the outputs from SVMs and they are selected by

D: Do you do SNS? **B: Ya.
A: Facebook and ...
C: Yup. I do.
D: I use Twitter. ++D: Ameba Pigg is a kind of SNS?
B: Well, I suppose it's a SNS.++
C: Meh, maybe.
A: There is something about Ameba Pigg in recent days.**
B: Huh? ... Happening?
C: I don't know.
A: You know, elementary school kid and junior high-school++
student were ...
A: They cracked passwords of some persons.
C: Ah! Ah! I remember now. ++C: The passwords were the date of birth or something...
B: Ya, ya!
B: Anyway, Mixi is SNS, isn't it? My first SNS is Mixi. **C: I agree.
B: Me too.
C: That is the encounter with SNS for many people.++B: I think a number of people do Mixi only.
D: Yup. ++D: Oh, Mobage, Mobage might be my first SNS. **B: I've not played Mobage.
A: Me too.
.........
Figure 4.

An example of the output from MethodP N M L .

the principle of the methodM L 7 . The utterances with “++”are the output by the methodP N . Therefore, Fig. 4 is an
output of the methodP N M L because it is the integration of
the methodM L and methodP N 8 .
In the experiment, three test subjects evaluated the outputs
from three methods and the baseline with the following
points; (1) readability, (2) understandable and (3) quantity ofthe summary. The evaluation point of “understandable” wasthe suitability of the context of summary. The evaluation
point of “quantity” was the adequacy of the size as thesummary. First, the test subjects received four summaries
by our methods simultaneously. Then, they compared thesummaries and judged the scores of each summary. The
score range from the test subjects was 1 (Bad) to 5 (Good)
points. We rank each method on the basis of the averagescore of each criterion.
Table III shows the ranking for each conversation. In
the ranking, the baseline and MethodP N M L were ranked
in the 1st place for three conversations and two conversations, respectively. Table IV shows the average ranking
for each method. In the evaluation score, there was a
7 Note that utterances with “**” are not all outputs from SVMs.
8 In other words, the combination of utterances with a rectangle andutterances with “++” is the methodP N and the combination of utteranceswith a rectangle and utterances with “**” is the methodM L .

Table IIIT HE RANKING OF EACH DATA .

ID
C1
C2
C3
C4
C5

1st
Baseline
Baseline
Baseline
MethodP N M L
MethodP N M L

2ndMethodP N M L
MethodP N M L
MethodP N M L
Baseline
Baseline

3rd
MethodP N
MethodP N
MethodP N
MethodM L
MethodM L

4th
MethodM L
MethodM L
MethodM L
MethodP N
MethodP N

Table IV
T HE AVERAGE RANKINGS OF EACH METHOD .
Baseline
1.4

MethodP N M L
1.6

MethodP N
3.4

MethodM L
3.6

large difference between the top two methods, namley thebaseline and MethodP N M L , and others, namely MethodP N
and MethodM L (1.4 and 1.6 vs. 3.4 and 3.6). Therefore, we
compared the top two methods in detail.
Table V shows the comparison of the baseline andMethodP N M L . In the table, RED and UND denote theevaluation criteria “readability” and “understandable”. The
baseline often generated high scores in the criteria as compared with the MethodP N M L . In the C4, the score of UND
for the baseline was better than that for the MethodP N M L
although the ranking of the MethodP N M L was better. The
reason was the length of the generated summaries. Table
VI shows the summary size of each method. In the table,
each cell denotes the number of utterances in the summary
and the summarization rates in parentheses. Since the sizeof the generated summaries by the baseline was the largest
in them, they contained much information. In other words,
the test subjects comprehensively judged that the generated
summaries by the MethodP N M L were suitable in terms of
the quality and the size. In other words, the MethodP N M L
was a balanced approach.
In addition, the summaries generated from the
MethodP N M L contained turning points in a conversation
and utterances with anaphoric relations. As a result, thesummaries were understandable although the size of themwas small as compared with the baseline method. On theother hand, one reason that the baseline method produced
good performance was that the method, namely SVMs,
included some context features, namely features between
utterances and anaphora. As a conclusion, information
about context is the most important for the conversation
summarization task. Therefore we need to apply othercontext features to the method for the machine learning
method and the integration process of the scoring andmachine learning.
In this paper, we focused on surface linguistic features
for the method. However, conversations contain many char-Table V
T HE COMPARISON OF BASELINE AND M ETHODP N M L .

ID
C1
C2
C3
C4
C5

Baseline
RED
UND
4.7
3.7
4.3
4.7
4.0
3.7
4.7
3.7
4.7
3.0

[1] James Allen and Mark Core. Draft of DAMSL: Dialog
act markup in several layers. Technical report, Universityof Rochester, Rochester, USA. The Multiparty DiscourseGroup., 1997.

MethodP N M L
RED
UND
3.3
3.0
3.0
3.0
3.3
3.3
3.7
3.7
4.7
3.7

[2] Harold P Edmundson. New methods in automatic extracting.
Journal of the ACM, 16(2):264–285, 1969.

Table VI
T HE SUMMARIZATION RATES . T HE NUMBER OF UTTERANCES ( THE
RATE ).
ID
C1
C2
C3
C4
C5
Ave

Baseline
60 (38.0)
79 (43.4)
63 (38.7)
66 (39.3)
72 (47.4)
68 (41.3)

MethodP N M L
49 (31.0)
56 (30.8)
57 (35.0)
57 (33.9)
52 (34.2)
54 (33.0)

MethodP N
38 (24.1)
44 (24.2)
45 (27.6)
45 (26.8)
42 (27.6)
43 (26.1)

R EFERENCES

MethodM L
30 (19.0)
32 (17.6)
36 (22.1)
33 (19.6)
31 (20.4)
32 (19.7)

acteristics, such as prosodic features [10], hot spots in eachconversation [12] and laughing [7]. In addition, conversations contain discourse information such as dialogue acts[1]. These characteristics are useful for the summarization
method. In particular, discourse information is important forthe summary generation process to select appropriate utterances. Incorporating them to our method is the important
future work.
V. C ONCLUSION
In this paper, we proposed a method for conversation
summarization. For the method, we utilized two approaches,
namely a scoring method and a machine learning technique,and integrated them in the final summary generation process.
In the experiment, the scoring method obtained highaccuracy. The weighting factor basing on the utterance
length was effective for the scoring. The method based on
SVMs also generated high performance. The features in thetarget utterance were the most effective for the method.
For the integration process, we proposed three approaches.
We compared our methods with a baseline method based on
SVMs only in the experiment. For the overall evaluation,
the ranking of our best method sometimes was lower thanthe baseline method. However, our method was a balanced
approach in terms of the quality and the size. In other words,
the summaries by our method were small and suitable ascompared with the baseline method.
Future work includes (1) introducing other context features, (2) eliminating redundant outputs from SVMs, and(3) applying new features, such as laughing information, to
our method.

[3] Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard
Pfahringer, Peter Reutemann, and Ian H. Witten. The weka
data mining software: An update;. In SIGKDD Explorations,volume 11, 2009.
[4] Ryuichiro Higashinaka, Yasuhiro Minami, Hitoshi Nishikawa,
Kohji Dohsaka, Toyomi Meguro, Satoshi Kobashikawa, Hirokazu Masataki, Osamu Yoshioka, Satoshi Takahashi, andGenichiro Kikui. Improving hmm-based extractive summarization for multi-domain contact center dialogues. In Spoken
Language Technology Workshop, pages 61–66, 2010.
[5] Inderjeet Mani. Automatic Summarization (Natural Language
Processing, 3). John Benjamins Pub Co, 2001.
[6] Hitoshi Nishikawa, Takaaki Hasegawa, Yoshihiro Matsuo, andGenichiro Kikui. Improving quality of extractive summarization via detecting intersentential anaphora. In Proceedings of
NLP2011 (in Japanese), 2011.
[7] Kazutaka Shimada, Akihiro Kusumoto, Takahiko Yokoyama,
and Tsutomu Endo. Hot spot detection in multi-party conversation using laughing feature. In Technical report of IEICE,NLC2012-7, pages 25–30, 2012.
[8] Wataru Sunayama and Masahiko Yachida. A panoramicview system for extracting key sentences with discovering
keywords featuring a document. Systems and Computers inJapan, 34(11):81–90, 2003.
[9] V. N. Vapnik. Statistical Learning Theory. Wiley, 1999.
[10] Shasha Xie, Dilek Hakkani-Tur, Benoit Favre, and Yang
Liu. Integrating prosodic features in extractive meetingsummarization. In IEEE Workshop on Automatic Speech
Recognition & Understanding, ASRU 2009., pages 387–391,
2009.
[11] Shasha Xie, Yang Liu, and Hui Lin. Evaluating the effectiveness of features and sampling in extractive meetingsummarization. In Spoken Language Technology Workshop,
pages 157–160, 2008.
[12] Takahiko Yokoyama, Kazutaka Shimada, and Tsutomu Endo.
Hot spot detection in multi-pary conversation using linguistic
and non-linguistic information. In Proceedings of NLP2012(in Japanese), 2012.
[13] Klaus Zechner. Fast generation of abstracts from generaldomain text corpora by extracting relevant sentences. In Proceedings of the 16th conference on Computational linguistics,
volume 2, pages 986–989, 1996.