International Journal of Service and Knowledge Management
International Institute of Applied Informatics
2020, Vol. 4, No. 1, 1 - 26

Creating Character Networks with Kinship Relations from
ComicsHarumi Murakami *, Yusuke Nagaoka *, Ryota Kyogoku *Abstract
To find and understand comic contents, we present an approach of creating character networks with semantic relations from comics using the panels and speech balloons. First, webuild a dataset that contains a comic’s characters and its words in speech balloons separated
by the panels. Second, we extract the semantic relations among characters using the words
in the speech balloons and panels. Third, we develop character networks by identifying theimportant characters using the frequencies of characters and their relatedness, and then we
add semantic relations to these networks. As an initial step in finding semantic relations,
we focus on kinship relations. We implement our approach and successfully build character networks from four popular Japanese comics. Finally, we evaluate our kinship relation
extraction algorithms and the created character networks.
Keywords: manga, comics, character networks, panels, speech balloons

1

Introduction

The popularity of manga (Japanese comics) has increased worldwide [1]. Many manga havebeen translated into other languages, and a large number of manga are being published.
For example, over 1 million copies of “Pokémon: The Electric Tale of Pikachu” issue#1 have been sold in the United States, making it the best-selling comic book there since
1993 [2]. Manga are usually serialized in magazines and later compiled in books. Somemanga resemble long epics. For example, the popular manga “Naruto” was serialized in
the magazine Weekly Shónen Jump from 1993 vol. 43 [3] to 2014 vol. 50 [4], later to becompiled into 72 books. A total of 700 episodes appear in the 72 books. Since finding an
episode, a volume (book) of a manga, or even a particular manga itself from a large-scalemanga collection is difficult, we investigate ways of finding a particular episode or volume(book) of a particular manga, as well as a manga series itself.
Character networks, which include character correlation charts or diagrams, are often
created to introduce the contents of TV dramas in Japan to viewers, thus helping them better
understand the characters and stories. Character networks for comics, movies, animation,novels, etc., can also be found on the web. These are created manually for the same purpose∗

Osaka City University, Osaka, Japan2

H. Murakami, Y. Nagaoka, R. Kyogoku

as the TV networks. We believe that such character networks are useful for finding and understanding the contents of comics. However, it is unclear how to most effectively createcharacter networks from comics. On the other hand, there exists much research on creating
character networks from literary texts such as novels. It is obvious that character names andsemantic relations between characters can be extracted from quotes or explanatory text using natural language understanding techniques. Since comics are not originally constructedfrom texts but from images in which text information is attached, it remains unclear how to
identify character names from images and how to extract dialogues from speech balloons;furthermore, there tends to be little explanatory text aside from that in the speech balloons.
This research creates character networks from comics to help users (1) find an episode, avolume of comics, or a comic itself and (2) understand the contents of episodes or volumesof comics, or the comics themselves. In this research we use the term comics instead of
manga because our research is also applicable to western comics.
In our previous work [5], we proposed a method of creating character networks using the frequencies of characters and their co-occurrences by referring to panels 1. Although that pioneering work highlighted the potential of such an approach, we did not implement algorithms for drawing character networks or present any for extracting the semantics among characters using texts. In this research that uses the words included in speechballoons, we enhance our previous work and create character networks with semantic relations. In addition, we revise our algorithm that lays out character networks and present adrawing algorithm that creates a network called an “overview character network” to display
important characters and overview the contents in a single diagram.
In what follows, related work is shown in Section 2. We explain our approach’s overview
and our algorithms in Section 3. Experimental results using four popular Japanese comics
are shown in Section 4. We discuss the significance of our research in Section 5.

2

Related WorkThe research on creating character networks can be categorized by the target, such ascomics, texts, and videos. Furthermore, the process of creating character networks canbe divided into two steps: (1) extracting relations (including semantic relations) between
characters and (2) drawing character networks. In this section, we start our discussion byfocusing on these two steps for each target.

2.1

Creating Character Networks from Comics

To the best of our knowledge, our previous work [5] with its original idea is the first researchthat created character networks from comics. In that work, we manually created a datasetthat includes characters divided by panels. We assumed that characters who appear in thesame or in the adjacent panels are related and calculated the relatedness between charactersusing co-occurrences in the same or adjacent panels. Finally, we presented a layout of character networks using the frequencies of characters and the relatedness between characters.
We used a volume of a famous Japanese comic (Dragon Ball vol. 32 [6]) in the experiment.
In this paper, we extend our previous work by (1) using words in speech balloons to
extract semantic relations between characters and (2) presenting new algorithms that
1 We used the term “frame” in the previous work but we changed it to “panel” since panel is more precise
for describing our work.

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

Creating Character Networks with Kinship Relations from Comics

create character networks with semantic relations. We implemented and evaluated our
new algorithms using four different popular Japanese comics: Naruto, Dragon Ball(different volumes from that used in the previous work), Boys Over Flowers, and SpaceBrothers. This paper is based on our conference papers [7] [8]. Detailed definitions o falgorithms, explanations, and further discussion have been added. In addition, newcharacter networks are presented.
After our work, [9] is the only research we know of that has created character networks
from comics. They presented a method of creating character networks based on the speechtiming of characters. They manually extracted pairs of a character name and its speechtiming from three Japanese comics (one volume for each comic). They calculated relatedness between two characters based on similarity of speech timing. Their networks did
not link characters directly but instead linked a character and a “group,” which is a conceptof grouped related characters. They did not use any panel information or text and did not
extract any semantic relations.
Since no other work has created character networks from comics, we discuss character
networks created from other media.

2.2 Creating Character Networks from Literary Texts
Much research has created character networks from literary texts such as novels. We briefly
discuss such research by selecting examples of them. The main differences between research on comics and that on literary texts are (1) research on literary texts extracts character names and relations (including semantic relations) automatically, and (2) it extracts
information from quoted text and/or body text.
Elson et al. [10] extracted social networks from 19th century British novels and serials
for literary studies. They found instances of quoted speech, attributed each quote to acharacter, and identified when certain characters are having a c onversation. This work is
related to ours in the sense that they create character networks based on the frequency of thecharacters and the interaction-based-weight between them. However, they did not display
any semantic relations among characters.
Nishihara [11] extracted the relations among characters from Japanese novels and created character networks. His work is the most closely related to ours among the research
efforts on texts because he deals with Japanese and creates character networks with semantic relations such as kinship. The main difference is that he deals with novels instead of
comics. Another major difference for connection generation is that he does not specify
characters to display; we choose main characters with related characters to summarize acomic’s content in one diagram. As for the extracting relation algorithms, he mainly extracts useful information from explanatory texts using pattern-matching. For example, the“P1 is a sister of P2” template extracts a P1-sister-P2 relation. He defined 5 3 relations
among characters including kinship relations. Since comics generally have few explanations within their text, we treat the text in speech balloons near the characters as dialogues
and extract kinship relations from the above information.
Kokkinakis et al. [12] identified personal relationships between main characters in 19th
century Swedish fiction using information guided by named e ntities. They used a relationship vocabulary, which includes kinship relations, for labeling and extracted sentences withco-occurring pairs of person named entities and labeled the extracted pairs of person entities using window sizes. One way is a window size of 1-3 tokens that uses pattern matching
templates with lexical units from the resources. Another way is with a window size of 4-10

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

3

4

H. Murakami, Y. Nagaoka, R. Kyogoku

tokens to measure the context similarity between the extracted pairs of person entities.
Our way is different from the work of Kokkinakis et al. in a sense that we treat the texts
in the speech balloons near the characters as dialogs, as we described in the previous
paragraph. In their later work, Kokkinakis et al. [13] created a simplified cooccurrence network for the characters in the Swedish novel “Vi Bookar, Krokar och
Rothar.” They did not add any semantic relations in the network.
Lee and Yeung [14] created people (characters) and place networks from literary texts.
They translated the first five books of the Hebrew bible into English and used them for theirevaluation. They used not only dialogue interaction but also explanatory texts as weight
relations between characters. They did not display any relations between characters. Their
networks can be distinguished from others in the sense that they drew place nodes.

2.3 Creating Character Networks from Other Media
Some research created character networks not from the contents themselves but from external texts. This text-based approach is applicable to multimedia contents including comics
and videos.
Goto et al. [15] created human correlation charts from TV program abstracts for movies
in Electronic Program Guides (EPG). Although the targets (dramas and movies) are multimedia, this research uses text as well as natural language understanding techniques. The
main feature of this study is that anaphora resolution is used to identify (merge) characters.
Their character network uses groups of words for semantic relations, not only nouns suchas “only daughter” but also verbs such as “loves,” “be married,” and “declare an affair.”Ramakrishna et al. [16] built character networks to analyze differences with respect tothe characters’ gender, race, age, and other metadata in movies. They obtained movie scripts
from the Internet along with additional metadata such as cast, genre, writers, and directors;
moreover, they collected actor-level demographic information such as gender, race and age.
Their demo of character network visualization is available on the web (https://sail.usc.edu/
˜mica/project pages/character network visualization.html). The shape and color of character objects vary according to gender and race. Important characters are computed by
network analysis, and links between characters are based on dialogues.
Spysee2 (http://spysee2.jp/), whose original algorithm was based on Matsuo et al. [17],extracted person information from the web and then displayed social networks. Famous
manga character names such as Naruto can be input. Users can explore the social networks
created from the information about Naruto available on the web, not from media contents.
Even though this method is useful for well-known comics, it is inadequate to express thecharacter networks within such designated units as episodes or volumes based on media
contents.
Although it is more challenging than using external texts, some research created character networks from videos, and we describe two examples of such research as follows.
Park et al. [18] built a network structure called Character-Net from Video. It finds characters in a group of shots, extracts the speaker and listeners in the scene, represents it withcharacter-based graphs and draws the relationship between all characters by accumulating
the character-based graphs at video. The characters are identified b y f ace r e c ognition. The
names of the characters can be detected using matching technology between subtitles andscript. The major characters are identified by network a n alysis. They used three videos fortheir experiment.

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

Creating Character Networks with Kinship Relations from Comics

Yuan et al. [19] presented a graph-based correlation search framework to analyze actor association, which enables semantic-level user navigation to better understand sitcom
contents. They developed a new retrieval mode, character correlations search, that is deployed in their VisualCar actor-concurrence search system. They performed face detection
and tracking on cluster faces. The clustered faces are manually adjusted for actor indexing.
They proposed a context-based correlation analysis in shot sequence and a hierarchical concurrence measurement for refinement. They used the “Friends” (20 hours) sitcom in theirexperiments.

2.4

Computational Research on Comics

We have discussed how to create character networks from comics and other media. As theremaining body of related work, we briefly give an overview of the computational researchon comics. Matsushita [20] called research on digital comics “comic computing,” and inanother work [21] selected 53 papers for classifying comic computing into three issues: (1)comic coding, (2) utilizing obtained knowledge, and (3) utilizing comic presentation. Our
previous work [5] was classified into a subcategory of ( 1): understanding the structure of
comic contents.
Using the current technology, it is impossible to extract character names with words
in speech balloons separated by panel information. In our research, we manually identified panels and speakers (characters) of speech balloons and created a d ataset. Ueno andKazama [9] also manually identified speech timing. If we consider the entire required technology as separate elements such as extracting panels or words in speech balloons, there
exists some previous research. For example, Pang et al. [22] proposed a robust method
for automatically extracting panels from digital manga pages. Guo et al. [23] extracted
text strings from comic strips. Chu and Li [24] detected faces in manga. We can use thesetechnologies as preprocessing to create a dataset. However, the difficult parts include identifying characters from images (some characters are drawn from the back view, without aface image), giving names to identified characters, and identifying speakers and listeners in
relation to speech balloons. At least in the immediate future, manual effort is inevitable increating a useful dataset. One possible solution is combining comic images with manual
annotation. Fujimoto et al. [25] created Manga109, a dataset covering a variety of Japanesecomic books, totaling 109, publicly available for academic purposes. This dataset provides
numerous comic images but lacks annotations, so they have presented web-based softwarefor efficient annotations.

3

Method3.1

Overview

This research creates character networks with semantic relations from comics using thepanels and speech balloons to find and understand comic contents. First, we build a datasetthat contains a comic’s characters and its words in speech balloons separated by panels.
Second, we extract the semantic relations among characters using the words in the speechballoons and the panels. Third, we develop character networks by identifying the importantcharacters using the frequencies of characters and their relatedness, and then we add semantic relations to the networks. An overview of our approach is shown in Figure 1. As an
initial step in finding semantic relations, we focus on kinship relations such as father-child

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

5

6

H. Murakami, Y. Nagaoka, R. Kyogoku

Comic

Master-Pupil *Father-Child
Like *Creating
DatasetDatasetCreating
Character
Networks

Extracting
Semantic
Relations

Overview Character Network

Kinship Relations
Without InferenceWith Inference* Future Work

Figure 1: Overview of our approach

relations. The process of extracting semantic relations is divided into two stages: one without inference and the other with inference. As a first example of character n etworks, we
present a network called an overview character network that displays important characters
to overview the comic’s contents in a single diagram. The main feature of this researchis creating character networks from comics: (a) identifying the semantic relations between
characters using the words in speech balloons and panels and (b) identifying the relatedness
between characters using their co-occurrences in the panels.
In the following, we illustrate our approach using the popular comic Naruto vols. 46[26] - 50 [27]. Note that the examples presented in this paper were translated into English
for publication.

3.2

Creating Dataset

We manually create a comic dataset comprised of sets of a character and his/her words inspeech balloons that are divided by panels.
Figure 2 shows an example of creating a dataset. Note that Japanese comics flow from
top to bottom and right to left. We identify panels and characters inside panels with words
in speech balloons and write them down in a dataset. In the dataset, the sequence of panels
is divided by “*”. A character with a speech balloon is written as “character name: words
in speech balloons”. A character without a speech balloon is written as “character name:”.
Character names that cannot be identified a re a ssigned a s A , B , C , . . . , f or c ounting thenumber of characters used in the algorithms.

3.3

Extracting Semantic Relations

We aim to extract semantic relations among characters. As an initial step, we focus on
kinship relations. In this research, we try to capture calls (such as ”father” or ”mother”)
using the words in speech balloons and panels. For example, if the words in speech balloons
contain the word father and satisfy certain conditions, the speaker must be a child. We
assume that the recipient of this call will be characters in the same or in the next panels.
If there is one or more characters in the same panel with the speaker, they are consideredcandidates of the call’s recipient and we give a point to them. If there is no character inthe same panel, we give a point to the characters in the adjacent (previous and next) panels.
In our preliminary experiment, we tried various patterns of giving a point to characters,

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

Creating Character Networks with Kinship Relations from Comics

Kakashi Hatake

First panel on page 8
Dad, hecame!!

Last panel on page 7

Speech balloon
Choza Akimichi

Choji Akimichi

NagatoPage 8(a) Comic

Page 7*
Nagato:*Choji Akimichi: Dad, he came!!
*
Kakashi Hatake:
Choza Akimichi:*(b) Dataset

Figure 2: Example of creating a dataset
such as in the same panel only, previous panel only, next panel only, the same and adjacentpanels, etc. Consequently, we chose the proposed algorithm because it provided the best
performance.

3.3.1

Extracting Semantic Relations using Kinship Dictionary

We illustrate our method with the following child-father relation.
Step 1. Matching whether a term in the dictionary (e.g., dad, father) appears in thewords.
Table 1 shows our dictionary with six family relations. Since there were few grandchildgrandparent relations in the dataset and the words ojiichan (grandpa) and obaachan (grandma)in Japanese are often used to call to elderly people (not actual grandparents), we just added
neutral terms for grandfather and grandmother. For similar reasons, we did not create a dictionary for person-uncle or person-aunt relations, since the words ojisan (uncle) and obsan(aunt) in Japanese are often used to call to adults (not actual uncles or aunts). We expected
to extract these four relations with inference processes. Likewise, words such as oniisan(brother) or oneesan (sister) are used to call to young adults. However, we decided to add
many terms to build person-older brother and person-older sister dictionaries, since siblings
are important in the dataset in our experiment. Note in Japan, although younger siblings use
brother or sister to denote their older brother/sisters, older siblings seldom use these terms
for younger brother/sisters.
Step 2. When a term exists in words, if the previous term is not no (means of) and thefollowing term is not a particle, we assume that the speaker is talking to his father.
Step 2.1 When there are other characters in the same panel as the speaker, a point isgiven to the semantic relations between the speaker and the other characters in the panel.
Step 2.2 When there is only one character (speaker) in the panel, a point is given to thesemantic relations between the speaker and other characters in the adjacent (previous andnext) panels.
In the example of Figure 2, since Dad is included in the child-father dictionary andcondition Step 2 is also satisfied, semantic relations are created between Choji Akimichi

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

7

8

H. Murakami, Y. Nagaoka, R. Kyogoku

Table 1: Kinship dictionary
RelationFrom

To

child-father
child-motherperson-older brotherperson-older sister
grandchild-grandfather
grandchild-grandmother

child
child
personpersongrandchildgrandchildfathermotherolder brother
older sister
grandfather
grandmotherNo. of
terms
13
13
17
17
1
11

Sample terms
dad, father
mom, motherbrothersister
grandfather
grandmotherand Nagato, Kakashi Hatake, and Choza Akimichi. In this case, Choji Akimichi who said
Dad becomes a child and Nagato, Kakashi Hatake, and Choza Akimichi become fathers.
Figure 3 shows how to calculate initial scores for child-father relations between (from- to) characters. Here, (a) corresponds to the real example in Figure 2. The speaker of the
speech balloon is Choji Akimichi. Since there is no other character with Choji Akimichi inthe second panel, 1 is added to characters in the first and third panels. On the other hand,(b) is an imaginary example used to explain what happens if Choza Akimichi exists in thesecond panel. In this case, 1 is added to only this character in the same panel with the
speaker Choji Akimichi.
Kakashi Hatake

From Character A to
Character B

1Dad, hecame!!

1

1
Choza Akimichi

Choji Akimichi

NagatoInitial Score for
Semantic Relation

From Choji Akimichi to
Nagato1

From Choji Akimichi to
Kakashi Hatake

1

From Choji Akimichi to
Choza Akimichi

1(a) Real example
Kakashi Hatake

Dad, hecame!!

From Character A to
Character B

1
Choza Akimichi

Choji Akimichi

NagatoInitial Score for
Semantic Relation

From Choji Akimichi to
Choza Akimichi

1

If Choza Akimichi existed here(b) Imaginary example

Figure 3: Example of calculating initial semantic relations

Table 2 shows the extracted semantic relations from Naruto vols. 46-50. Twelve childfather and eight child-mother relations were extracted.

3.3.2

Identifying Spouse and Modifying Semantic Relations

In Japan, married couples with children often call each other father or mother as if they
are nicknames. We also use this custom for our work. If two characters call each otherfather and mother, they are identified as a married couple, a new married couple relation

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

9Creating Character Networks with Kinship Relations from Comics

Table 2: Extracted semantic relations using kinship dictionary
No.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

Character 1
Choji Akimichi
Choji Akimichi
Choji Akimichi
Kakashi Hatake
Shima
Shima
Shima
Naruto Uzumaki
NagatoNagatoOnoki
Onoki
Kiba Inuzuka
Kiba Inuzuka
Kiba Inuzuka
Kiba Inuzuka
Fukasaku
Fukasaku
NagatoNagatoCharacter 2
NagatoKakashi Hatake
Choza Akimichi
Sakumo Hatake
Fukasaku
NagatoNaruto Uzumaki
Minato Namikaze
Nagato’s father
Nagato’s mother
Danzo
Gaara
Akamaru
Tsume Inuzuka
Kuromaru
Katsuyu
Naruto Uzumaki
Shima
Nagato’s father
Nagato’s mother

Relationchild-father
child-father
child-father
child-father
child-father
child-father
child-father
child-father
child-father
child-father
child-father
child-father
child-motherchild-motherchild-motherchild-motherchild-motherchild-motherchild-motherchild-mother

Score
3
2
4
3
4
3
3
3
1
1
1
1
2
2
2
1
2
4
1
1

(in this paper, a wife-husband relation) is created and the previous child-father and childmother relations are deleted. In other words, if character 1 and character 2 have child-father
relation points and character 2 and character 1 have child-mother relation points, character1 must be a wife and character 2 must be a husband.
In the example in Table 2, a wife-husband relation between Shima and Fukasaku is created. The original two Shima between Fukasaku (child-father and child-mother) relations
are deleted.

3.3.3

Modifying Child-Parent Relations

In general, a person has just one father and one mother. We select the most likely childparent relations based on the relations.
If a person has multiple child-father or child-mother relations, the relations are selectedwith maximum scores. If multiple relations remain, we perform pattern-matching on thecharacter names and select the relation that has the most similar name. If the similarity isidentical, all of the relations are deleted.
From three potential relations when the child is Choji Akimichi in Table 2, we selected
Choji Akimichi and Choza Akimichi as the child-father relation, since this relation’s scoreis the maximum (4) while the other two child-father relations were deleted.
There are two child-father relations in which the child is Shima. Since their scores are
identical, the name similarity for each relation is calculated. The similarity is also identical

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

10

H. Murakami, Y. Nagaoka, R. Kyogoku(0), so the two relations were deleted. Likewise, we deleted two child-father relations for thechild Nagato, two child-father relations for the child Onoki, and two child-mother relations
for the child Nagato.
There are four child-mother relations whose child is Kiba Inuzuka. First, Kiba InuzukaKatsuyu is deleted because its score is 1 (not maximum). Next, name similarity chooses
Tsume Kinuzuka as Kiba Inuzuka’s mother.
We obtained the result semantic relations shown in Table 3.
Table 3: Extracted semantic relations after section 3.3.3
No.
1
2
3
4
5
6

3.3.4

Character 1
Choji Akimichi
Kakashi Hatake
Naruto Uzumaki
Kiba Inuzuka
Fukasaku
Shima

Character 2
Choza Akimichi
Sakumo Hatake
Minato Namikaze
Tsume Inuzuka
Naruto Uzumaki
Fukasaku

Relationchild-father
child-father
child-father
child-motherchild-motherwife-husband

Score
4
3
3
2
2
–

Modifying Semantic Relations using Thresholds

Since the main and related characters appear many times, many (relatively) unimportant
relations might be extracted. We set thresholds to delete such relations. We want to define
a function Tc,2 (ci , c j ) that obtains a threshold between two characters ci and c j .
First, we classified the characters into five groups: 1, 2, 3, 4, and 5 for the top 20%, 40%,
60%, 80%, and other, respectively, according to their cumulative appearance rates. Whena character’s rate lies in multiple groups, the character is classified into the upper group.
In our Naruto example, there are a total of 8,089 character appearances. Naruto Uzumaki(the main character) appears 1,210 times, and his appearance rate is 15% (1, 210/8, 089).
Nagato appears 1,200 times, and his cumulative appearance rate is between 15% and 30%((1, 210 + 1, 200)/8, 089). These two characters are classified in group 1 (see Table 4).
Next we set thresholds for each group.
max(score)− (k − 1),2
where g is a group and k is the group number (i.e., 1, 2, 3, 4 or 5).
If T (gk ) is under 0, then set zero to Tgroup (gk ):(round up to pos integer if T (gk ) > 0,
Tgroup (gk ) =0otherwise.
T (gk ) =(1)

For our Naruto example, max(score) for extracted relations is 4 (see Table 2). The
threshold for group 1 is 2, for group 2 it is 1, and for groups 3 to 5 it is 0 from Eq. (1).
Third, we set thresholds for each relation using the minimal character threshold:Tc,1 (ci ) := Tgroup (gk ),where ci is a character included in the group gk (see examples of Tc,1(ci) in Table 4).
Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

11

Creating Character Networks with Kinship Relations from Comics

Table 4: Ranking of character appearances
Rank
1
2
3
4
5
5
7
8
9
10
11
12
13
···
26
27
28
28
30
31
32
···

Character
Naruto Uzumaki
NagatoSakura Haruno
Sasuke Uchiha
Kakashi Hatake
A
Tobi
Sai
Onoki
Danzo
Fukasaku
Konan
Killer Bee···
YahikoJugo
Jiraiya
YamatoShizuneMinato Namikaze
Shikamaru Nara···

Appearance
1,210
1,200
264
253
251
251
202
172
167
154
152
141
140
···
88
86
84
84
83
81
76
···

Group
1
1
2
2
2
2
3
3
3
3
3
3
3
···
4
4
4
4
4
4
4
···

Threshold
2
2
1
1
1
1
0
0
0
0
0
0
0
···
0
0
0
0
0
0
0
···

Display
X
X
X
X

X

X
X
···
X
X

X
X
···

Finally, we set the threshold between two characters ci and c j to minimal thresholds forci and c j as shown in Eq. (2):Tc,2 (ci , c j ) = min(Tc,1 (ci ), Tc,1 (c j )).

(2)

For example, for the Naruto Uzumaki-Minato Namikaze child-father relation, the threshold for Naruto Uzumaki is 2 and for Minato Namikaze is 0, and the threshold for theirrelation becomes 0. Since the score of this relation (3) exceeds the threshold (0), it is notdeleted.
In our example, since the scores of all the examples exceed their threshold, no relation
is deleted in Table 3.

3.3.5

Adding Semantic Relations using Inference

Family inference is a typical example for artificial intelligence. Such relations as grandparentgrandchild can be defined using basic parent-child or spouse relations. Table 5 shows theinference rules used in this research. With the above rules, person-uncle and person-aunt
relations can be generated without the kinship dictionary.
In our Naruto example, from the Naruto Uzumaki-Minato Namikaze child-father relation and the Fukasaku-Naruto Uzumaki child-mother relation, a Fukasaku-Minato Namikaze

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

12

H. Murakami, Y. Nagaoka, R. Kyogoku

Table 5: Inference rules
A father of a father is a grandfather.
A mother of a mother is a grandmother.
A father of a mother is a grandfather.
A mother of a mother is a grandmother.
People who share a father are brothers or sisters.
People who share a mother are brothers or sisters.
A brother/sister of a father is an uncle/aunt.
A brother/sister of a mother is an uncle/aunt.
Brothers or sisters have identical parents.

grandchild-grandfather relation can be created.
Table 6: Extracted final semantic relations and precision judgment
No.
1
2
3
4
5
6
7

Character 1
Choji Akimichi
Kakashi Hatake
Naruto Uzumaki
Kiba Inuzuka
Fukasaku
Shima
Fukasaku

Character 2
Choza Akimichi
Sakumo Hatake
Minato Namikaze
Tsume Inuzuka
Naruto Uzumaki
Fukasaku
Minato Namikaze

Relationchild-father
child-father
child-father
child-motherchild-motherwife-husband
grandchildgrandfather

Judge
X
X
X
X
× errorX
× error (withinference only)

Precision: 5/6 = 0.833 (without inference); 5/7 = 0.714 (with inference)
Table 6 shows the final result for our Naruto example. We extracted six relations without
inference and added one relation by the inference process. Five of six relations (without
inference) are correct and out of seven (with inference) are correct.

3.4 Creating Character Networks
Various networks can be drawn, and we present an overview character network that displays
important characters in a single diagram. This new algorithm is different from the algorithm
presented in our previous work [5].
We assume that critical characters appear more frequently than other characters. We
select the three most frequent characters (we call them “main” characters 2 ) first and thendisplay three related characters for them. Finally, lines and semantic relations are drawn
among characters. Creating an overview character network algorithm is described below.
Step 1. Three “main” characters whose appearance is in the top three are displayed up,
down-left, and down-right.
2 We distinguish main characters and “main” characters in this paper.

The main characters are the original meaning and the “main” characters are the three characters displayed at center in the overview character
networks.

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

13

Creating Character Networks with Kinship Relations from Comics

Step 2. Three other related characters whose relatedness (Eq. (3), see below) is in the
top three for the “main” characters (except for the “main” characters) are selected. Theseother characters are displayed next to the three “main” characters. When they are related tomore than one “main” character, they are displayed between the related “main” characters.
Step 3. Lines are connected from the “main” characters to the related other characters.
Step 4. Lines are connected between the “main” characters whose relatedness are over
the threshold 3 .
Step 5. The relation names are described next to the relations. We use Father-Childfor child-father relations, Mother-Child for child-mother relations, Married Couple forwife-husband relations, Sibling for person-older brother, person-older sister, brother or sister relations, Grandfather-Grandchild for grandchild-grandfather relations, GrandmotherGrandchild for grandchild-grandmother relations, Uncle-Nephew/Niece for person-uncle
relations, and Aunt-Nephew/Niece for person-aunt relations.
Calculating the relatedness among characters, which was originally presented in ourprevious work [5], is redefined as follows. Here, we want to define a function R(ci , c j ) that
calculates a relatedness score between characters ci and c j . The relatedness score between
characters ci and c j is calculated as the summation of their related scores using the coappearance inside the panels and the scores using the appearance in the adjacent panels, as
shown in Eq. (3). The basic ideas are that (1) characters who exist in the same panel arevery related, (2) characters who exist in an adjacent panel are somewhat related, and (3) the
relatedness decreases according to the number of characters in the same or in an adjacentpanel.
First, a function Rinside (ci , c j ) defines the inside scores:(
1/nRinside (ci , c j ) =0.25

if n = 1, 2, 3,
if n ≥ 4,

where n is the number of different characters from the designated character ci inside apanel.
Next, a function Rad jacent (ci , c j ) defines the adjacent 4 scores:(
0.5/n if n = 1, 2, 3, 4,
Rad jacent (ci , c j ) =0.1
if n ≥ 5,
where n is the number of different characters from the designated character ci in theadjacent panel.
Finally, R(ci , c j ) defines the related scores:
m

(k)

m−1

(k,k+1)R(ci , c j ) = ∑ Rinside (ci , c j ) + ∑ Rad jacent (ci , c j ),
k=1

(3)

k=1

where the summation occurs between the designated panels. For example, if threepanels are given, Rinside is done three times (first, second, and third panels), and Rad jacent isdone twice (between first and second panels and between second and third panels).
3This threshold is not the same as that described in Section 3. In this research, we set 3 to the threshold fordrawing by checking the relatedness data. We need to evaluate this in the future.
4We used the term “next” in the previous work but we changed it to “adjacent” since adjacent is more precise
for describing our work.

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

14

H. Murakami, Y. Nagaoka, R. Kyogoku

Figure 4 shows how to calculate relatedness scores between characters using the same
examples in Figure 3. In the real example (a), the relatedness between Kakashi Hatake andChoza Akimichi is 1 (1 + 0), since these two are in the same panel once. In the imaginary
example, the relatedness score between Kakashi Hatake and Choza Akimichi (described as
Choza Akimichi and Kakashi Hatake in order of appearance in Figure 4, which is identicalsince there is no direction of relatedness) is 1.5: 1 (between two characters in the third
panel) + 0.5 (between Choza Akimichi in the second panel and Kakashi Hatake in the third
panel).
Kakashi Hatake
0.25
1Dad, hecame!!

0.5

0.25
Choza Akimichi

Choji Akimichi

Nagato(a) Real example

Kakashi Hatake

Between Character A
and Character B

Relatedness
Score

Nagato and Choji Akimichi

0+0.5=0.5

Choji Akimichi and Kakashi
Hatake0+0.25=0.25

Choji Akimichi and Choza
Akimichi0+0.25=0.25

Kakashi Hatake and Choza
Akimichi

1+0=1

0.25
0.25
1

0.5
0.25

Choza Akimichi

Dad, hecame!!
1

Choji Akimichi

0.25

NagatoIf Choza Akimichi existed here(b) Imaginary example

Between Character A
and Character B

Relatedness
Score

Nagato and Choza Akimichi

0+0.25=0.25

Nagato and Choji Akimichi

0+0.25=0.25

Choza Akimichi and Choji
Akimichi

1+0.25=1.25

Choza Akimichi and Kakashi
Hatake1+0.5=1.5

Choji Akimichi and Kakashi
Hatake0+0.25=0.25

Figure 4: Exmple of calculating relatedness

After removing incorrect semantic relations from the extracted results manually, an
overview character network is created for Naruto (Figure 5 5 ). Naruto Uzumaki (main character), Nagato, and Sakura Haruno are displayed as the three “main” characters. Since
Kakashi Hatake is selected as being among the top three with Naruto Uzumaki and Sakura
Haruno, he is displayed between them. No line is drawn between Nagato and Sakura
Haruno because their relatedness is below the threshold. Father-Child relation name is displayed alongside the line between Naruto Uzumaki and Minato Namikaze. Four infrequentcharacters, as categorized in group 4 (Yahiko, Jiraiya, Minato Namikaze, and Shikamaru
Nara), are displayed in the character network from their relatedness to the “main” characters. Table 4 lists the ranking of character appearances, and Table 7 ranks the relatedness
between characters. Please check these tables for the characters and relations (marked as
Display in the final column) that appear in Figure 5.

4

Experiment

We successfully built overview character networks from four Japanese comics. In this section, we first focus on the evaluation of algorithms for extracting semantic (kinship) relations. Next, we analyze the generated overview character networks.
5 In this paper’s figures, a space inside names is replaced by an underbar for implementation reasons.

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

Creating Character Networks with Kinship Relations from Comics

Figure 5: Overview character network created for Naruto vols. 46-50

4.1

MethodWe built four datasets from four comics: Naruto, vols. 46-50, Dragon Ball (kanzenban)
vols. 29 [28] - 34 [29], Boys Over Flowers, vols. 16 [30] - 20 [31], and Space Brothers,
vols. 1 [32] - 5 [33]. The reasons for selecting these comics are as follows: (1) they containa certain amount of kinship relations, (2) they cover various comic types by readers: boys
(Naruto, Dragon Ball), girls (Boys Over Flowers), and youth (Space Brothers), and (3)they cover various sections of comics: opening (Space Brothers), middle (Naruto, Boys
Over Flowers), and closing (Dragon Ball). The average number of total characters (fourcharacters in Figure 2) was 10,985. Furthermore, 6,581 characters (one character in Figure2) out of these 10,985 had speech balloons and were identified as speakers (60%).
In all four comics, We identified the following semantic relations and their amounts: 22
child-father, 18 child-mother, 11 wife-husband, 4 person-older brother, 2 person-older sister, 2 grandchild-grandfather, 1 grandchild-grandmother, and 1 person-uncle. We identified
no person-aunt relations.
We evaluated our algorithms without and with inferences. The following are the evaluation measures for extracting semantic relations:
Precision =Recall =number of correct extracted relations,
number of extracted relations

number of correct extracted relations,
number of correct relations to be extracted
F1 =2 × Precision × Recall.
Precision + Recall

For our Naruto example, Table 6 shows how to calculate Precision and Table 8 shows
how to calculate Recall. We generated six kinship relations, five of which were correct
Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

15

16

H. Murakami, Y. Nagaoka, R. Kyogoku

Table 7: Ranking of character relatedness
Rank
1
2
3
4
5
6
6
8
9
10
11
12
13
14
15
16
17
···
52
···
62
···

Character 1
Naruto Uzumaki
Naruto Uzumaki
NagatoNaruto Uzumaki
Naruto Uzumaki
Killer Bee
Naruto Uzumaki
NagatoNagatoNaruto Uzumaki
Naruto Uzumaki
Sasuke Uchiha
Naruto Uzumaki
Karui
Sakura Haruno
Sasuke Uchiha
Nagato···Kakashi Hatake···
Sakura Haruno···

Character 2
NagatoSakura Haruno
Konan
Fukasaku
Minato Namikaze
Kisame Hoshigaki
Kakashi Hatake
YasukeJiraiya
Katsuyu
Sai
A
Karui
Omoi
Sai
Karin
Kakashi Hatake···
Sakura Haruno···
Shikamaru Nara
···

Score
442.292
98.65891.40091.34285.68383.91766.62562.85858.76755.29252.56751.26751.21747.30843.35837.18335.433···17.550···14.558···

Display
X
X
X
X
X
X
X
X

X

···
X
···
X
···

without inference, so the precision without inference is 0.833. Seven (one more) relations
were generated with inference, five of which were correct, so the precision with inferenceis 0.714 (Table 6). Eight kinship relations exist in the dataset, and five were extracted, so
the recall is 0.625 (Table 8).

4.2
4.2.1

Results and Analysis
Extracting Semantic Relations Algorithms

Table 9 shows the experimental results for semantic (kinship) relations without and withinference for each comic. On average, precision is 0.500 (with) to 0.570 (without), recall is0.447 (without) to 0.520 (with), and F1 is 0.440 (with) to 0.461 (without). Overall, without
inference is better in precision and F1 and with inference is better in recall. Dragon Ballis an exception, where with inference is better in all measures. Among the four comics,
the results for Naruto are the best. We believe this is because the number of relations issmallest (8) and all relations (child-father, child-mother, and wife-husband) are relatively
easy to extract. On the other hand, the results for Dragon Ball are the worst. We believe
this is because the number of relations is the largest (24) and various types of semantic
relations, such as grandchild-grandfather, grandchild-grandmother, person-older brother,and person-uncle, are included. In addition, we observed that there exist fewer speech

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

Creating Character Networks with Kinship Relations from Comics

Table 8: Correct semantic relations to be extracted and recall judgment
No.
1
2
3
4
5
6
7
8

Character 1
Choji Akimichi
Kakashi Hatake
Naruto Uzumaki
Kiba Inuzuka
Shima
NagatoNagatoIno Yamanaka

Character 2
Choza Akimichi
Sakumo Hatake
Minato Namikaze
Tsume Inuzuka
Fukasaku
Nagato’s father
Nagato’s motherInoichi Yamanaka

Relationchild-father
child-father
child-father
child-motherwife-husband
child-father
child-motherchild-father

Judge
X
X
X
X
X
× not extracted
× not extracted
× not extracted

Recall: 5/8 = 0.625 (without and with inference)balloons than in other comics and that the characters tend to call each other by name rather
than by dictionary terms such as father or mother. We assume that these features affected
the results. Furthermore, we used the closing part of Dragon Ball to create its dataset. We
assume that there is less information on semantic relations in the closing part than in theother parts of comics, since such information has already been explained in the first half ofthe story.
From the inference process, 25 relations were created. Unfortunately, 22 relations wereincorrect and only three relations were created correctly: Son Goten-Son Goku child-father
relation (Dragon Ball), Tsukushi Makino-Haruo Makino child-father relation (Boys Over
Flowers), and Susumu Makino-Chieko Makino child-mother relation (Boys Over Flowers).
In addition, Tsukasa Domyouji-Tsubaki Domyouji sibling relation was created (Boys Over
Flowers). In Table 9, we counted as correct a person-older sister relation in convenience.
This is because sibling relation and person-older brother/sister relatsions are redundant andsibling relation is excluded from the evaluation.
Table 10 shows the results for each semantic relation. Since there exist few results
except for child-father, child-mother, wife-husband, person-older brother, and person-older
sister relations, we analyzed just these five r elations. For the child-father and child-motherrelations, without inference is better in precision and F1 while with inference is better inrecall. For wife-husband relations, nothing was added in the inference process, and theresults were identical. Person-older brother and person-older sister relations proved muchworse than child-father and child-mother relations in precision. We believe this is because(1) words such as brother and sister are more frequently included than father or mother in
NOT call context and (2) there are few selection processes for relations other than thresholdcompared to child-father or child-mother relations.
Using our Naruto example, we explain the following three types of errors: precision,
recall, and inference.
The precision error occurs when there are two characters within the same or adjacentpanels and kinship terms (terms in a kinship dictionary) are included in a speech balloon.
Under this condition, the errors can be classified into the following types: (a) When the extraction (call) is not correct (e.g. calling a man using brother, calling oneself using father), a
wrong relation is created. (b) When the recipient of a call is incorrect, an incorrect relationis created. (c) When a wrong relation is chosen among the relation candidates. (d) When
there exist one or more children with a father and/or a mother call, a wrong wife-husband

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

17

18

H. Murakami, Y. Nagaoka, R. Kyogoku

Table 9: Results of semantic relations for each comic
Comic
NarutoDragon Ball
Boys Over Flowers
Space Brothers
Average

Without inference
Prec
Rec
F1
0.833
0.625 0.714( 5/ 6) ( 5/ 8)
0.667
0.167 0.267( 4/ 6) ( 4/24)
0.368
0.583 0.452( 7/19) ( 7/12)
0.412
0.412 0.412( 7/17) ( 7/17)
0.570
0.447 0.461

With inference
Prec
Rec
F1
0.714
0.625 0.667( 5/ 7)( 5/ 8)0.714
0.208 0.323( 5/ 7) ( 5/24)
0.303
0.833 0.444(10/33) (10/12)
0.269
0.412 0.326( 7/26) ( 7/17)
0.500
0.520 0.440

Table 10: Results of semantic relations for each semantic relation
Relationchild-father
child-motherwife-husband
person-older brotherperson-older sister
grandchild-grandfather
grandchild-grandmotherperson-uncle
person-aunt

Without Inference
Prec
Rec
F1
0.688
0.500 0.579(11/16) (11/22)
0.750
0.333 0.462( 6/ 8) ( 6/18)
0.500
0.364 0.421( 4/ 8) ( 4/11)
0.182
0.500 0.267( 2/11) ( 2/ 4)0.200
0.500 0.286( 1/ 5)( 1/ 2)0.000
0.000 0.000( 0/ 0)( 0/ 2)0.000
0.000 0.000( 0/ 0)( 0/ 1)0.000
0.000 0.000( 0/ 0)( 0/ 1)0.000
0.000 0.000( 0/ 0)( 0/ 0)With Inference
Prec
Rec
F1
0.500
0.591 0.542(13/26) (13/22)
0.412
0.389 0.400( 7/17) ( 7/18)
0.500
0.364 0.421( 4/ 8) ( 4/11)
0.182
0.500 0.267( 2/11) ( 2/ 4)
0.333
1.000 0.500( 2/ 6)( 2/ 2)0.000
0.000 0.000( 0/ 3)( 0/ 2)0.000
0.000 0.000( 0/ 1)( 0/ 1)0.000
0.000 0.000( 0/ 1)( 0/ 1)0.000
0.000 0.000( 0/ 0)( 0/ 0)

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

Creating Character Networks with Kinship Relations from Comics

Table 11: Results for semantic relations in overview character networks for each comic
Comic
NarutoDragon Ball
Boys Over Flowers
Space Brothers
Average

Full-automatic
Prec
Rec
F1
0.500 1.000 0.667( 1/ 2) ( 1/ 1)
1.000 1.000 1.000( 2/ 2) ( 2/ 2)0.250 1.000 0.400( 1/ 4) ( 1/ 1)
0.500 0.500 0.500( 1/ 2) ( 1/ 2)
0.563 0.875 0.642

Semi-automatic
Prec
Rec
F1
1.000 1.000 1.000( 1/ 1) ( 1/ 1)
1.000 1.000 1.000( 2/ 2) ( 2/ 2)
1.000 1.000 1.000( 1/ 1) ( 1/ 1)
1.000 0.500 0.667( 1/ 1) ( 1/ 2)
1.000 0.875 0.917

Full automatic contains without/with inference results.
relation is created. In Naruto, Fukasaku and Naruto Uzumaki appear together many times
in the same panels. Fukasaku calls his wife Shima “mother,” but more points were given
to Naruto Uzumaki, and thus the Fukasaku-Naruto Uzumaki child-mother relation was extracted. This is the type (d) error.
The recall errors can be classified into four types: (a) When two characters do notappear within the same or adjacent panels. (b) When there is no speech balloon between
two characters. (c) When the extraction (call) fails (e.g. kinship terms are not included).
(d) When a wrong relation is chosen or nothing is chosen among relations. We failed to
extract Nagato-Nagato’s father child-father, Nagato-Nagato’s mother child-mother, and Ino
Yamanaka-Inoichi Yamanaka child-father relations. Moreover, there was only one “father
call” and one “mother call” from Nagato to other characters. In these cases, Nagato’s father
and mother were in the same panel, but our algorithms could not identify which person
was the father or the mother. These are type (d) errors. For the Yamanaka case, there wasno “father call” in the dataset. This error is type (c). We did not find any case where the“When the recipient of a call is wrong” error (type (b) precision error). We believe that for
our algorithm our basic idea, i.e. “giving an initial point to characters in the same panel andwhen there is no other character in the same panel giving it to characters in the adjacentpanels,” is reasonable.
All inference errors occurred from incorrect relations. As an example in Naruto, fromthe Naruto Uzumaki-Minato Namikaze child-father relation and the Fukasaku-Naruto Uzumaki child-mother relation, a Fukasaku-Minato Namikaze grandchild-grandfather relation
was created. One apparent solution would be for people to check and correct their answers
before the inference process and then make inferences based on correct relations.
From the precision and recall failure analysis, we identified room for improvement andthe following possible solutions: (a) add terms to the dictionary (for grandchild-grandparent,
person-uncle/aunt relations, in particular); (b) modify the initial extraction algorithms toproperly capture the calls; (c) modify the algorithms of selecting relations; (d) extract relations using explanatory text. We assume that (b) and (d) will greatly improve the results
by using more careful natural language understanding processing. For example regarding(b), we can add such a condition as “when a kinship term is followed by tachi (plural), no
relation should be created.” For example regarding (d), from such a speech balloon as “Oh,Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

19

20

H. Murakami, Y. Nagaoka, R. Kyogoku

Figure 6: Overview character network created for Dragon Ball vols. 29-34
yes! Hibito is my younger brother!” the Hibito Nanba-Mutta Nanba person-older brotherrelation can be extracted.

4.2.2

Created Character Networks

We attempted (a) full-automatic display (without inference), (b) full-automatic display(with inference), and (c) semi-automatic display (display after removing wrong semantic relations). Table 11 shows the results on narrowing down the target to the semantic relations
to be drawn in the overview character networks. The full-automatic results contain bothwithout and with inference cases; in other words, the results are the same in full-automatic
displays (a) and (b). Precision is 0.563, recall is 0.875, and F1 is 0.642 in full-automatic displays, and precision is 1.000 (naturally), recall is 0.875, and F1 is 0.917 in semi-automatic
display. Since limited semantic relations are drawn in overview character networks, theresults are much higher than those in Table 9.
When we used full-automatic display (without/with inference), five wrong labels weredisplayed: Fukasaku-Naruto Uzumaki child-mother (Naruto), Shigeru Okawahara-Shigeru’s
mother child-father, Akira Mimasaka-Tsukasa Domyouji person-older brother, Tsukushi
Makino-Rui Hanazawa person-older brother (Boys Over Flowers), and Reiji Nitta-Yasushi
Furuya person-older brother (Space Brothers) relations. One recall error was not extracting the Hibito Nanba-Mutta Nanba person-older brother relation (Space Brothers). Dragon
Ball showed perfect results. When we used semi-automatic display, Naruto and Boys Over
Flowers attained perfect results as well as Dragon Ball.
We analyze the four networks in detail by referring to Figures 5, 6, 7, and 8, which werecreated by (c) semi-automatic display.
Naruto belongs to the genre of battle manga. Naruto is the main character. Naruto
vols. 46-50 tell the story of the fight between Naruto and Nagato. Naruto and Nagato are
Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

Creating Character Networks with Kinship Relations from Comics

Figure 7: Overview character network created for Boys Over Flowers vols. 16-20

Figure 8: Overview character network created for Space Brothers vols. 1-5

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

21

22

H. Murakami, Y. Nagaoka, R. Kyogoku

displayed as “main” characters. Their teacher is the same, Jiraiya. Nagato, Konan, andYahiko form a group of isolated enemies who fight against Naruto (Figure 5). Nagato and
Sakura Haruno are almost always in different places and thus they are not linked together.
Dragon Ball can also be classified as battle manga. Son Goku is the main character.
Dragon Ball (kanzenban) vols. 29-34 tell the story of Majin Boo. In Figure 6, Son Goku
and Pure Evil Majin Boo, who is one of the different forms of Majin Boo, are displayed
as “main” characters. In this figure, Pure Evil Majin Boo is the only enemy, and the othercharacters except for Videl and Kibito fought him.
Boys Over Flowers is a romantic comedy manga. The main character is Tsukushi
Makino. The second main character is Tsukasa Domyouji, a boy who loves her. Rui
Hanazawa, Sojiro Nishikado, and Akira Mimasaka are third-level main characters, and they
and Tsukasa Domyouji are called “F4 (flower-four).” Boys Over Flowers vols. 16-20 involve the story of a triangular relationship among Tsukushi Makino, Tsukasa Domyouji,
and Shigeru Okawahara (Tsukasa’s fiance). Figure 7 shows the love triangle of Tsukushi
Makino, Tsukasa Domyouji and Shigeru Okawahara as “main” characters. Nakatsuka is anenemy in this network while the other characters are good friends. Here, F4 members arealso displayed.
Space Brothers is a narrative manga. Mutta Nanba and Hibito Nanba are brothers, as
the first and second main characters, who aim to become astronauts. Space Brothers vols.
1-5 tell the story of Mutta Nanba challenging himself toward becoming an astronaut alongwith other candidates. In Figure 8, the “main” characters Mutta Nanba, Yasushi Furuya,
and Serika Ito are on the same team in the astronaut program. Naoto Fukuda is also on thesame team and linked to all three “main” characters. Reiji Nitta is on the same team too.

5

Discussion

Finally, we discuss the contributions of our research. First, to the best of our knowledge,
our work (including our previous work) is the only research that creates character networks
with semantic relations from comics. The main feature of this research is creating character
networks from comics using the words in the speech balloons and panels. Second, we
evaluated our algorithm for extracting kinship relations and found that without inference isbetter in precision and F1 and with inference is better in recall. Third, we implemented ouralgorithms and successfully built overview character networks from four popular Japanesecomics: Naruto, Dragon Ball (using different volumes from our previous work), Boys Over
Flowers, and Space Brothers.
Even though basic algorithms using panels are applicable to any language, some heuristics of extracting semantic relations depend on culture and language. For example, forwife-husband relation extraction, we used Japanese married couples with children who refer to each other as father and mother. We must consider heuristics for other cultures andlanguages.
Here, we describe future work. First, we need to improve the extraction of kinshiprelation algorithms. We have already suggested ideas for improvement in Section 4.2.1.
Second, we need to evaluate overview character networks in more detail. Third, we need toconsider other semantic relations, such as “friends” and “rivals,” to make character networks
more useful. As Figure 1 shows, “likes” might be a one-way relation, and thus the type of
relations should be considered in the future.
We have already presented overview character networks, but we could also use our
Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

Creating Character Networks with Kinship Relations from Comics

Figure 9: Single-focus character network created for Tsukasa Domyouji in Boys Over
Flowers, vols. 16-20

Figure 10: Frequent co-occurrence character network created for Boys Over Flowers, vols.
16-20

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

23

24

H. Murakami, Y. Nagaoka, R. Kyogoku

approach to create other possible types of networks: (a) single-focus character networks
and (b) frequent co-occurrence character networks. Next, we explain our semi-automatic
examples using Boys Over Flowers to show a different example than Naruto.
If users wanted to focus on one character, a network centering on that one character
would be useful. Figure 9 shows such an example of a “single-focus character network.”
Kaede Domyouji (mother) and Tsubaki Domyouji (sister) are linked from Tsukasa Domyouji (second main character in Boys Over Flowers). Our algorithm allows users to set link
depths, and breadth-first search is applied to each link depth using relatedness. If a user
selects Tsukasa Domyouji and displays the single-focus character network with the entire
link depth mode, all characters directly related from Tsukasa Domyouji are linked first, andthen Sakurako Sanjo, Kiyonosuke Amakusa, and Shizuka Todou are linked from Tsukushi
Makino, who is the most related to Tsukasa Domyouji.
If users wanted to find a certain story or character, networks for frequent co-occurrence
characters using relatedness are useful. Figure 10 shows an example of a user inputting
100 as threshold value for relatedness and displaying frequent co-occurrence character networks. The screen shows characters whose relatedness is over 100. In this case, there is
only one network centering on the main character Tsukushi Makino. The user can recall aside story between Tsukushi Makino and Tama, or another story between Yuuki Matsuoka
and Sojirou Nishikado. The users can input threshold values as they like, enabling them tocheck the story in depth.
A variety of other networks are possible, and these should be investigated in the future.

6

Conclusions

To find and understand comic contents, we presented an approach of creating character networks with semantic relations from comics using the panels and speech balloons. First, webuild a dataset that contains a comic’s characters and its words in speech balloons separated
by the panels. Second, we extract the semantic relations among characters using the words
in the speech balloons and panels. Third, we develop character networks by identifying theimportant characters using the frequencies of characters and their relatedness, and then we
added semantic relations to these networks. As an initial step in finding semantic relations,
we focused on kinship relations.
We implemented our approach and successfully built overview character networks that
display important characters to overview the comic’s contents in a single diagram. This was
carried out for four popular Japanese comics: Naruto, Dragon Ball, Boys Over Flowers,
and Space Brothers. Finally, we evaluated our kinship relation extraction algorithms andthe created overview character networks.
Future work includes improving the extracting kinship relation algorithms, evaluating
the created overview character networks in more detail, considering additional semantic
relations such as friends and rivals, and creating different types of networks.

References[1] W. S. Wang, “Globalizing manga: From Japan to Hong Kong and beyond,”
Mechademia, pp. 23–45, 2006.
Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

Creating Character Networks with Kinship Relations from Comics[2] J. J. Miller, “The last million-selling comic book in North America?
It’s Batman vs. Pokémon for the title,” 2014. [Online]. Available:
http://blog.comichron.com/2014/05/batman-vs-pokemon-last-million-selling.html[3] Weekly Shónen Jump.

Shueisha (in Japanese), 1993, vol. 43.

[4] Weekly Shónen Jump.

Shueisha (in Japanese), 2014, vol. 50.

[5] H. Murakami, R. Kyogoku, and H. Ueda, “Creating character connections frommanga,” in Proc. 3rd Int. Conf. Agents and Artificial Intelligence (ICAART 2011),
vol. 1, 2011, pp. 677–680.
[6] A. Toriyama, Dragon Ball.

Shueisha (in Japanese), 1992, vol. 32.

[7] Y. Nagaoka, R. Kyogoku, and H. Murakami, “Creating character connections fromcomics using frames and words in balloons,” in Human Communication Group Symposium 2015, 2015, pp. 12–13, (in Japanese).
[8] H. Murakami, Y. Nagaoka, and R. Kyogoku, “Creating character networks fromcomics using frames and words in balloons,” in Proc. 2018 7th IIAI Int. Congr. Advanced Applied Informatics (IIAI-AAI 2018), 2018, pp. 1–6.
[9] T. Ueno and K. Kazama, “Method of creating character networks based on speechtiming in comics,” in Proc. 11th ARG SIG-WI2, 2017, pp. 1–6, (in Japanese).
[10] D. K. Elson, N. Dames, and K. R. McKeown, “Extracting social networks from literary fiction,” in Proc. 48th Annu. Meeting Association for Computational Linguistics(ACL 2010), 2010, pp. 138–147.
[11] H. Nishihara, “Extraction of relations among characters in literary texts,” Master’s
Thesis, JAIST, 2015, (in Japanese).
[12] D. Kokkinakis and M. Malm, “Character profiling in 19th century fiction,” in Proc.
Language Technologies for Digital Humanities and Cultural Heritage Workshop,
2011, pp. 70–77.
[13] D. Kokkinakis, M. Malm, J. Bergenmar, and A. Ighe, “Semantics in storytelling in
Swedish fiction,” in DATeCH ’14 Proc. 1st Int. Conf. on Digital Access to Textual
Cultural Heritage 2014, 2014, pp. 137–142.
[14] J. Lee and C. Y. Yeung, “Extracting networks of people and places from literary texts,”
in Proc. 26th Pacific Asia Conf. Language, Information, and Computation (PACLIC
2012), 2012, pp. 209–218.
[15] J. Goto, Y. Yagi, A. Aizawa, and S. Sekine, “Generation of correlation charts fromTV programs based on anaphora resolution,” in 22nd Annu. Conf. Japanese Society
for Artificial Intelligence, 2008, (in Japanese).
[16] A. Ramakrishna, V. Martinez, N. Malandrakis, K. Singla, and S. Narayanan, “Linguistic analysis of differences in portrayal of movie characters,” in Proc. 55th Annu. Meeting Association for Computational Linguistics (ACL 2017), 2017, pp. 1669–1678.
Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.

25

26

H. Murakami, Y. Nagaoka, R. Kyogoku[17] Y. Matsuo, J. Mori, M. Hamasaki, K. Ishida, T. Nishimura, H. Takeda, K. Hashida,
and M. Ishiduka, “Polyphonet: An advanced social network extraction system,” inProc. WWW2006, 2006, pp. 397–406.
[18] S.-B. Park, Y.-W. Kim, M. Uddin, and G.-S. Jo, “Character-Net: Character network
analysis from video,” in WI-IAT ’09 Proc. 2009 IEEE/WIC/ACM Int. Joint Conf. Web
Intelligence and Intelligent Agent Technology, vol. 1, 2005, pp. 305–308.
[19] K. Yuan, R. Ji, H. Yao, X. Sun, P. Xu, and X. Liu, “VisualCor System: Search actor
correlations in TV series,” in ICIMCS ’09: Proc. 1st Int. Conf. Internet Multimedia
Computing and Service, 2009, pp. 213–218.
[20] M. Matsushita, “The possibility of comic computing,” in Proc. 2nd ARG WI2, no. 15,
2013, (in Japanese).
[21] M. Matsushita, “The past and future of comic computing,” in Proc. 11th JSAI Interactive Information Access and Visual Mining, 2015, pp. 12–19, (in Japanese).
[22] X. Pang, Y. Cao, R. W. H. Lau, and A. B. Chan, “A robust panel extraction method
for manga,” in MM ’14 Proc. 22nd ACM int. conf. Multimedia, 2014, pp. 1125–1128.
[23] Q. Guo, K. Kato, N. Sato, and Y. Hoshino, “An algorithm for extracting text strings
from comic strips,” in SIGGRAPH ’06 ACM SIGGRAPH 2006 Research posters,
no. 76, 2006.
[24] W.-T. Chu and W.-W. li, “Manga FaceNet: Face detection in manga based on deepneural network,” in ICMR ’17 Proc. 2017 ACM Int. Conf. Multimedia Retrieval,no. 76, 2017, pp. 412–415.
[25] A. Fujimoto, T. Ogawa, K. Yamamoto, Y. Matsui, T. Yamasaki, and K. Aizawa,“Manga109 dataset and creation of metadata,” in MANPU ’16 Proc. 1st Int. Workshop on coMics Analysis, Processing and Understanding, no. 2, 2016.
[26] M. Kishimoto, Naruto.

Shueisha (in Japanese), 2009, vol. 46.

[27] M. Kishimoto, Naruto.

Shueisha (in Japanese), 2010, vol. 50.

[28] A. Toriyama, Dragon Ball (Kanzenban).

Shueisha (in Japanese), 2004, vol. 29.

[29] A. Toriyama, Dragon Ball (Kanzenban).

Shueisha (in Japanese), 2004, vol. 34.

[30] Y. Kamio, Boys Over Flowers.

Shueisha (in Japanese), 1997, vol. 16.

[31] Y. Kamio, Boys Over Flowers.

Shueisha (in Japanese), 1998, vol. 20.

[32] C. Koyama, Space Brothers.

Kodansha (in Japanese), 2008, vol. 1.

[33] C. Koyama, Space Brothers.

Kodansha (in Japanese), 2009, vol. 5.

Copyright © by IIAI. Unauthorized reproduction of this article is prohibited.