# Classification of Social Media Messages Posted at the Time of Disaster

Kemachart Kemavuthanon, Osamu Uchida

► To cite this version:

Kemachart Kemavuthanon, Osamu Uchida. Classification of Social Media Messages Posted at the Time of Disaster. 4th International Conference on Information Technology in Disaster Risk Reduction (ITDRR), Oct 2019, Kyiv, Ukraine. pp.212-226, 10.1007/978-3-030-48939-7_18 . hal-03374229

HAL Id: hal-03374229

https://inria.hal.science/hal-03374229

Submitted on 12 Oct 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Classification of Social Media Messages
## Posted at the Time of Disaster

Kemachart Kemavuthanon<fnref n="1" /> and Osamu Uchida<fnref n="2" />

**Abstract.** Nowadays, social media is one of the essential sharing of information and proliferation tools because it spreads text messages, news, pictures, or videos in real-time. During the disaster, Japanese people use social media to exchange real-time information for their social interaction. Twitter is the most popular tool that has been used for disaster response in Japan. Even though many disaster systems have been created and used for disaster mitigation in Japan, most of them are assumed to be used by the Japanese in the Japanese language. From this problem, this study focuses on the way to create a disaster response system and community service to help, collect, and extract information on social media to help disaster mitigation becomes more important. This paper aims to investigate the tweets by focusing on noun keywords during the Osaka North Earthquake on 18 June 2018 with a data set of more than 9,000,000 tweets. The process presented classify social media messages by using ontology, word similarity, frequency of keyword, and evaluate results of natural language processing. We organize the messages into 15 categories and used as the classification algorithms with machine learning features of the count of each category word in the sentences. The result tweets were statistically compared with the keyword in each category to classify the content and collecting disaster information and using the result to build the analysis system.

**Keywords:** Disaster Information, Word Similarity, Twitter Analysis, Tweet Classification, Natural Language Processing, Neural Disaster

## 1 Introduction

During the disaster, Japanese people utilize social media to exchange useful information in real-time. For example, during the 10 minutes from 8 o'clock immediately after the 2018 Osaka Northern Earthquake occurred, more than 270,000 tweets including the word “地震” (earthquake, in Japanese) were posted [1]. Even though many systems to use at the time of disaster have been created and used in disaster mitigation in Japan, most of them focused only on the Japanese people. Moreover, most information on social media during disasters does not help foreigners because the contents are written in Japanese. Therefore, we have been working on developing a system for foreigners in Japan, which is useful for obtaining necessary information in real-time

<footnote n="1">¹Graduate School of Science and Technology, Tokai University, Kanagawa, Japan</footnote>

<footnote n="2">²Dept. Human and Information Science, Tokai University, Kanagawa, Japan</footnote>

<footnote>kemachart.kem@mfu.ac.th, o-uchida@tokai.ac.jp</footnote>
during disasters. We use Twitter to gather information to be provided in our system because Twitter is the most utilized social media in Japan, with more than 45 million active users, and it is known that there are many tweets posted at the time of disasters[2][3].

In this study, we propose a method to classify tweet data using WordNet as a step of developing a disaster information providing system. In the experiment, we used a dataset of more than 9 million tweets collected on June 18, 2018, the day when the Osaka North Earthquake occurred. We verify the accuracy of the proposed tweet classification method by calculating the confusion matrix.

## 2 Related work

Lots of methods for the analysis of social media to create a disaster victims helping system. Several of them are explained in the following:

Disaster information tweeting system (DITS) and Disaster information mapping system (DIMS) is the application to share disaster information and help the user when disaster happens by use geolocation information and hashtag[4]. This system implemented as a web-based application. This application has unique features such that Tweets are posted as tweets from the user's own Twitter account, the user can send rescue information with the user's current geolocation information (the longitude and latitude coordinates) and share information between the users with texts and images. This application was launched in 2015 and the number of users of it is gradually increasing.

DETSAApp is the applied research for disaster events by summarization of images on Twitter[5]. The proposed method in [5] has the following features: (1) image clustering process with a near-duplicate image detection algorithm, and (2) image summarization using textual information associated with each image. That possesses the ability to portray the real-time scenario of an ongoing disaster event accurately.

Sumalatha et al. proposed an emergency distress relief system using social networking platform, called GDSS (Geo Distributed Social Service System) to provide immediate assistance [6]. People can upload the picture and/or image taken at the time of incident once they come across disaster, using the mobile in social media to the system. The system informs the nearest relief center and people at the nearest place to provide service and to take measures for recovery.

DISAANA and D-SUMM are the systems that are using Twitter as an information source to analyze AI and be used to create a help system in a disaster event [7]. DISAANA provides a list of answers to questions as to location and information. D-SUMM summarizes the disaster reports from a specified area in a compact format and enables rescue workers to grasp the disaster situations quickly. In the 2016 Kumamoto Earthquake, DISAANA used by the Japanese government and provided a wide range of useful information. It shows the overall information of the earthquake by choosing from keywords and related words as a layer of information to find the answers that are most closely related to the question.
However, in the research to create a helping system for disaster, there are still many things that need to be considered and developed. Especially in Japan, there is an overwhelming lack of research and development on systems that provide disaster information for foreigners.

# 3 Methodology

Many types of study for sentiment classification use machine learning [8]. Based on these studies, we propose a method to extract disaster information from social media data. The first step of the classification process consists of conducting a few necessary pre-processing steps, i.e., tokenization and removal of stop words [9]. Next, we select ten keywords to create a category that relates to the requirement for surviving during a disaster based on the recommendation of well-known Japanese information. After that, we compute the ontology and the WordNet similarity between each word in the tweet sentence and category keywords to find the meaning of vocabulary. Then, we classify the tweet sentence by using the frequency of keyword matching with the category.

## 3.1 Word similarity on WordNet

WordNet is a broad coverage lexical network of the English words, is organized into taxonomic hierarchies. Nouns, verbs, adjectives, and adverbs are divided into different groups named [10]. The process of computing the ontology and the WordNet similarity uses a library Java API for WordNet Similarity [11]. This process supports the semantic similarity between keyword and present similarity score with a percentage [12]. The lexical relations of WordNet include: the upper and lower position, synonyms, contains the property, causes [13][14]. WordNet similarity equation compares words by finding the root word of both words with function HyperTrees(). For example, the root word of cat: HyperTrees(Keyword:cat) = ROOT\*\#n\#1 < entity\#n\#1 < physical\_entity\#n\#1 < object\#n\#1 < whole\#n\#2 < living\_thing\#n\#1 < organism\#n\#1 < Keyword:animal < chordate\#n\#1 < vertebrate\#n\#1 < mammal\#n\#1 < placental\#n\#1 < carnivore\#n\#1 < carnine\#n\#2 < Keyword:cat. The results of both words were kept in the parameters T1 and T2. Then, it calculates the depth of the similarity by equation Eq. (1).

$$ 
\begin{aligned}
DepthLCS &= Depth(\text{Keyword}) \\
Depth1 &= \min\left(\text{depth}\left(\{\text{tree in T1}\}\middle|\text{tree contains LCS}\right)\right) \\
Depth2 &= \min\left(\text{depth}\left(\{\text{tree in T2}\}\middle|\text{tree contains LCS}\right)\right)
\end{aligned}
\qquad (1) $$

After that, we have a depth tree and results of T1 and T2, and we can calculate the similarity score by a use equation Eq. (2).

$$ 2 \times \frac{DepthLCS}{(Depth1 + Depth2)} \qquad (2) $$
From the calculating process with the WordNet similarity equation, we will get a sim-
ilar score for each word when comparing. The result will be 0-1, where 1 means 100%
related similar between two words and 0 will mean they are not similar at all.

3.2 Confusion Matrix

A confusion matrix is often used to measure the accuracy rate of the classifier. It uses
new data that is not used in the training process of the machine learning model [15][16].
The confusion Matrix has the following four values:

True Positive (TP): Both the prediction result and the actual class is true.

True Negative (TN): Both the prediction result and the actual class is not true.

False Positive (FP): The prediction result is true but the actual class is not true.

False Negative (FN): The prediction result is true but the actual class is true.

The measurement will measure all three things: accuracy, precision, and recall.

Accuracy: The value that presents the accurate ratio of the prediction,

$$
\frac{\text{TP}+\text{TN}}{\text{TP}+\text{TN}+\text{FP}+\text{FN}} \qquad (3)
$$

Precision: The value that indicates the correct answer rate when the prediction is true.

$$
\frac{\text{TP}}{\text{TP}+\text{FP}} \qquad (4)
$$

Recall (True Positive Rate): The value that indicates how much of the true class can be predicted correctly.

$$
\frac{\text{TP}}{\text{TP}+\text{FN}} \qquad (5)
$$

In this research, we use the confusion matrix to measure the accuracy of the prediction.

4 Data Collection and Analysis

4.1 Dataset Gathering

In this study, firstly, we derive a dataset from the previous process. It was tokenizing
[17] from the Japanese sentences of the Osaka North Earthquake on June 18, 2018,
such as; noun, verb, adverb, adjective, emoji, hashtag, link, and @Addfriend [18].

These APIs can also be used to access Twitter data [19] and the data from API is
JSON String file [20]. Then, all of the keywords have been translated into English to
know the meaning and understanding of each keyword by google translate API [21]. In
the second process, the result has been selected for analysis only 149,938 unique noun keywords and classify sentences. (The results are shown in **Error! Reference source not found..**)

<table><caption>Table 1. The top 20 most word noun.</caption><thead><tr><th>No</th><th>Word</th><th>Count</th><th>No</th><th>Word</th><th>Count</th></tr></thead><tbody><tr><td>1</td><td>地震 (earthquake)</td><td>12,116,336</td><td>11</td><td>水 (water)</td><td>667,229</td></tr><tr><td>2</td><td>大阪 (Osaka)</td><td>3,045,645</td><td>12</td><td>県 (Prefecture)</td><td>637,582</td></tr><tr><td>3</td><td>震度 (Seismic intensity)</td><td>2,409,063</td><td>13</td><td>熊本 (Kumamoto)</td><td>592,687</td></tr><tr><td>4</td><td>時 (Time)</td><td>1,841,154</td><td>14</td><td>余震 (aftershock)</td><td>588,481</td></tr><tr><td>5</td><td>北部 (North)</td><td>1,399,607</td><td>15</td><td>関西 (Kansai)</td><td>547,898</td></tr><tr><td>6</td><td>府 (Prefecture)</td><td>1,216,496</td><td>16</td><td>南部 (South)</td><td>536,719</td></tr><tr><td>7</td><td>情報 (information)</td><td>836,798</td><td>17</td><td>京都 (Kyoto)</td><td>494,031</td></tr><tr><td>8</td><td>発生 (Occurrence)</td><td>811,308</td><td>18</td><td>電車 (Electric train)</td><td>491,368</td></tr><tr><td>9</td><td>速報 (Breaking news)</td><td>693,661</td><td>19</td><td>緊急 (emergency)</td><td>478,020</td></tr><tr><td>10</td><td>震源 (Epicenter)</td><td>688,903</td><td>20</td><td>注意 (Caution)</td><td>466,612</td></tr></tbody></table>

## 4.2 Word similarity and frequency process

To analyze the data, we selected ten words from the word groups that have emerged from this study. They are necessary to know when a disaster happens. The category focuses on events- before the disaster happens, during the disaster happens and after a disaster occurs. During the process of comparing word similarities, we have analyzed keyword categories in more than ten categories. However, after the translation process and word similarity process, the result found that some categories have the same meaning and similar keyword content. Then, we decided to summarize tweet data into ten categories (see **Error! Reference source not found.**); transportation (as a group of travel information, vehicles, roads), animal (as a group of living, human, animal, and pets), alert (as a group of information during and after the disaster happens), warning (a group of caution and self-defense before a disaster), place (a group of building or locations), damage (a group of effects and violence by disasters), emotion (a group of feeling information and ideas), action (a group of activities during the disaster), energy (a group of energy information), service (a group of helping information and sharing service).

Then, Japanese nouns have been translated into English and filter the words that have the same meaning to reduce the number of keywords. The results of the English word compare the similarities and calculate the score of word similarity between words using WordNet and counting the frequency of those words in each sentence (9,428,334 tweets) to classify the type of sentence.
<figure><img src="image_2.png" /><figcaption>**Fig. 1.** The overall process of tokenizing, computing word similarities and counting word frequencies.</figcaption></figure>

## 4.3 Analysis of words frequency in message

From the Japanese message on Twitter, such as “駅が地震で壊れたため、大阪の電車は停車しました” (The Meaning is “The train in Osaka stopped because the station has broken by the earthquake”) to separate into each word. Then, all of the noun word grouping into ten categories of words based on their meaning. After that, the sentence will be counted frequency of the word in each category to rank the score (percent). The result of the score will present the main topic, the meaning of the sentence, and the rating of frequency used to classify messages to each category (see **Error! Reference source not found.**). However, several messages can be more than one category. It depends on the frequency score of the messages.

When the frequency score of each sentence has reached, we have verified and evaluated all the results by confusion matrix (measuring with Accuracy / Precision / Recall). The results of the process will compare to the real meaning of the actual content.

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** The process of deriving word frequencies of each Twitter message.</figcaption></figure>
# 5 Result

In this section, we describe the result of experiments to evaluate the accuracy of the feature sets described in Section 4 and the evaluation measured by the precision, recall, and accuracy. These experiments have been done on datasets of 9,428,334 tweets including the word “地震” (earthquake in Japanese) on the event the Osaka North Earthquake on June 18, 2018, that collected by using Twitter API.

Based on the results of the separation of Japanese message, many Japanese keywords have similar meanings when entering the translating process using Google Translate API; for example, Japanese keywords “電車,” “列車,” “トレーン,” and “汽車,” all mean “train” in English. We create relations of the database to link Japanese keywords and English keywords and also figure out which word comes from which sentence in the next process (see **Error! Reference source not found.**). As a result, Japanese 149,938 noun words become 59,236 English words. That used to find the similarity of the words in the next process.

When the result derived from the translation process and reduce the number of repeated meanings, the result keyword has to store in the database with the table linked to the original word table because all result should be able to connect to the tweet sentence. In the next step, we will compare the similarities and ontology of the 59,236 unique words. The number of results from all comparisons is 59,236 × 10 = 592,360 records. All calculation results from the word similarity process have stored in the database. In this regard, no matter how high or low the score is, we have to analyze results that can be used for the next research.

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** The translation process between Japanese noun keywords to English keywords using Google Translate API.</figcaption></figure>
<figure><img src="image_5.png" /><figcaption>**Fig. 4.** Percentage of unique words per group (Left) and percentage of using times in message per group (Right).</figcaption></figure>

Word similarity between all keyword with each category was calculated by using WordNet. For all noun keywords, we calculate the similarity score to ten categories. The total of keyword similarity in ten groups is more than 5,500 unique words and have been divided into the group of place 1,103 unique words (used 6,335,988 times in message), the group of transport 1,097 unique words (used 2,958,987 times in message), the group of animal 458 unique words (used 615,510 times in message), the group of alert 119 unique words (used 896,842 times in message), the group of warning 121 unique words (used 728,237 times in message), the group of damage 431 unique words (used 2,562,554 times in message), the group of emotion 176 unique words (used 226,251 times in message), the group of action 1,361 unique words (used 1,776,493 times in message), the group of energy 188 unique words (used 718,467 times in message) and the group of service and help 125 unique words (used 862,523 times in message). The number of keywords from the result in each category is derived from the comparison between keywords and ten categories to find the closest similarity rate, synonyms, and homonyms. The average result rate of each category depends on the ability to take advantage of the next process to find information. The scores of similarity rate are between 0.85 - 1.00 (see **Error! Reference source not found.**).

The word's similarity scores immediate difference in each category keyword due to the program find the result of the similarity based on the relationship and ontology of words. We have to analyze and select the most appropriate score in each category because a low similarity score that means, the word is not related and cannot filter the keyword of the sentence as we want. However, if we set too high a similarity score, the result will also lose that useful word. Therefore, the result has to determine the appropriate average rating from the keywords that can be analyzed and used in the content with the configuration as follows; the group of place: 0.80, the group of transport: 0.80, the group of animal: 0.70, the group of alert: 0.70, the group of warning: 0.75, the group of damage 0.75, the group of emotion: 0.75, the group of action: 0.75, the group of energy: 0.75, and the group of service and help: 0.75.
We have collected the top ten keywords of each category at the time of the earthquake to analyze the use of the term to find important information by differentiating the closest similarity rate (Error! Reference source not found.).

<table><caption>Table 2. The result top 10 most word noun in each category by similarity score.</caption><thead><tr><th colspan="4">Transport</th><th colspan="4">Place</th><th colspan="4">Animal</th></tr><tr><th>No</th><th>Word</th><th>Count</th><th>Ratio</th><th>No</th><th>Word</th><th>Count</th><th>Ratio</th><th>No</th><th>Word</th><th>Count</th><th>Ratio</th></tr></thead><tbody><tr><td>1</td><td>train</td><td>1,072,648</td><td>0.94</td><td>1</td><td>prefecture</td><td>1,216,496</td><td>0.94</td><td>1</td><td>cat</td><td>393,510</td><td>0.70</td></tr><tr><td>2</td><td>line</td><td>374,747</td><td>0.90</td><td>2</td><td>home</td><td>462,851</td><td>1.0</td><td>2</td><td>man</td><td>36,754</td><td>0.75</td></tr><tr><td>3</td><td>traffic</td><td>132,538</td><td>0.87</td><td>3</td><td>line</td><td>374,747</td><td>0.85</td><td>3</td><td>dog</td><td>17,661</td><td>0.75</td></tr><tr><td>4</td><td>release</td><td>46,207</td><td>0.82</td><td>4</td><td>work</td><td>226,212</td><td>0.80</td><td>4</td><td>head</td><td>9,085</td><td>0.70</td></tr><tr><td>5</td><td>delivery</td><td>35,071</td><td>0.80</td><td>5</td><td>station</td><td>218,792</td><td>1.0</td><td>5</td><td>horse</td><td>8,062</td><td>0.81</td></tr><tr><td>6</td><td>turn</td><td>31,748</td><td>0.89</td><td>6</td><td>target</td><td>216,486</td><td>1.0</td><td>6</td><td>bird</td><td>5,109</td><td>0.81</td></tr><tr><td>7</td><td>return</td><td>31,041</td><td>0.80</td><td>7</td><td>place</td><td>178,781</td><td>1.0</td><td>7</td><td>insect</td><td>3,504</td><td>0.80</td></tr><tr><td>8</td><td>transfer</td><td>29,027</td><td>1.0</td><td>8</td><td>area</td><td>127,034</td><td>0.94</td><td>8</td><td> creature</td><td>1,686</td><td>1.0</td></tr><tr><td>9</td><td>car</td><td>26,668</td><td>0.87</td><td>9</td><td>city</td><td>105,725</td><td>1.0</td><td>9</td><td>chicken</td><td>1,080</td><td>0.73</td></tr><tr><td>10</td><td>transport</td><td>25,572</td><td>1.0</td><td>10</td><td>center</td><td>100,094</td><td>0.80</td><td>10</td><td>beast</td><td>779</td><td>1.0</td></tr></tbody></table><table><thead><tr><th colspan="4">Damage</th><th colspan="4">Emotion</th><th colspan="4">Energy</th></tr><tr><th>No</th><th>Word</th><th>Count</th><th>Ratio</th><th>No</th><th>Word</th><th>Count</th><th>Ratio</th><th>No</th><th>Word</th><th>Count</th><th>Ratio</th></tr></thead><tbody><tr><td>1</td><td>intensity</td><td>2,409,063</td><td>0.93</td><td>1</td><td>feeling</td><td>74898</td><td>0.90</td><td>1</td><td>life</td><td>238,769</td><td>0.97</td></tr><tr><td>2</td><td>death</td><td>50,181</td><td>0.87</td><td>2</td><td>fear</td><td>37299</td><td>0.92</td><td>2</td><td>work</td><td>226,212</td><td>0.94</td></tr><tr><td>3</td><td>loss</td><td>27,632</td><td>0.96</td><td>3</td><td>love</td><td>34264</td><td>0.93</td><td>3</td><td>light</td><td>101,960</td><td>0.95</td></tr><tr><td>4</td><td>change</td><td>26,627</td><td>0.81</td><td>4</td><td>hate</td><td>13314</td><td>0.93</td><td>4</td><td>power</td><td>91,402</td><td>0.90</td></tr><tr><td>5</td><td>injury</td><td>25,192</td><td>1.0</td><td>5</td><td>care</td><td>8930</td><td>0.90</td><td>5</td><td>force</td><td>67,929</td><td>0.92</td></tr><tr><td>6</td><td>damage</td><td>24,584</td><td>1.0</td><td>6</td><td>panic</td><td>4796</td><td>0.93</td><td>6</td><td>weather</td><td>48,568</td><td>0.75</td></tr><tr><td>7</td><td>cost</td><td>16,292</td><td>0.75</td><td>7</td><td>spirit</td><td>2894</td><td>0.90</td><td>7</td><td>sun</td><td>81,411</td><td>0.82</td></tr><tr><td>8</td><td>price</td><td>16,292</td><td>0.94</td><td>8</td><td>emotion</td><td>839</td><td>0.90</td><td>8</td><td>heat</td><td>32,47</td><td>0.82</td></tr><tr><td>9</td><td>break</td><td>15,918</td><td>0.89</td><td>9</td><td>joy</td><td>614</td><td>0.90</td><td>9</td><td>energy</td><td>28,03</td><td>1.0</td></tr><tr><td>10</td><td>harm</td><td>6,283</td><td>1.0</td><td>10</td><td>concern</td><td>585</td><td>0.93</td><td>10</td><td>electricity</td><td>12,06</td><td>0.76</td></tr></tbody></table><table><thead><tr><th colspan="4">Alert</th><th colspan="4">Warning</th></tr><tr><th>No</th><th>Word</th><th>Count</th><th>Ratio</th><th>No</th><th>Word</th><th>Count</th><th>Ratio</th></tr></thead><tbody><tr><td>1</td><td>caution</td><td>466612</td><td>0.78</td><td>1</td><td>caution</td><td>466612</td><td>0.90</td></tr><tr><td>2</td><td>preparation</td><td>209549</td><td>0.92</td><td>2</td><td>rumor</td><td>215501</td><td>0.85</td></tr><tr><td>3</td><td>alarm</td><td>93425</td><td>1.0</td><td>3</td><td>advice</td><td>30863</td><td>0.92</td></tr><tr><td>4</td><td>signal</td><td>24030</td><td>0.70</td><td>4</td><td>report</td><td>11804</td><td>0.90</td></tr><tr><td>5</td><td>notification</td><td>22277</td><td>0.85</td><td>5</td><td>account</td><td>3097</td><td>0.90</td></tr><tr><td>6</td><td>wake</td><td>2222</td><td>0.80</td><td>6</td><td>lesson</td><td>1728</td><td>0.93</td></tr><tr><td>7</td><td>horn</td><td>885</td><td>1.0</td><td>7</td><td>recommendation</td><td>1425</td><td>0.88</td></tr><tr><td>8</td><td>sign</td><td>692</td><td>0.70</td><td>8</td><td>threat</td><td>1225</td><td>0.91</td></tr><tr><td>9</td><td>indication</td><td>342</td><td>0.87</td><td>9</td><td>comment</td><td>176</td><td>0.87</td></tr><tr><td>10</td><td>threat</td><td>225</td><td>0.90</td><td>10</td><td>example</td><td>159</td><td>0.93</td></tr></tbody></table>
<table><thead><tr><th colspan="4">Action</th><th colspan="4">Service</th></tr><tr><th>No</th><th>Word</th><th>Count</th><th>Ratio</th><th>No</th><th>Word</th><th>Count</th><th>Ratio</th></tr></thead><tbody><tr><td>1</td><td>operation</td><td>320172</td><td>0.88</td><td>1</td><td>work</td><td>226212</td><td>0.91</td></tr><tr><td>2</td><td>stop</td><td>126677</td><td>0.94</td><td>2</td><td>use</td><td>98831</td><td>0.78</td></tr><tr><td>3</td><td>fire</td><td>125801</td><td>0.93</td><td>3</td><td>force</td><td>67929</td><td>0.91</td></tr><tr><td>4</td><td>cause</td><td>110637</td><td>0.90</td><td>4</td><td>company</td><td>64728</td><td>0.90</td></tr><tr><td>5</td><td>end</td><td>73924</td><td>0.93</td><td>5</td><td>staff</td><td>26410</td><td>0.75</td></tr><tr><td>6</td><td>case</td><td>61262</td><td>0.90</td><td>6</td><td>service</td><td>25210</td><td>1.0</td></tr><tr><td>7</td><td>release</td><td>46207</td><td>0.90</td><td>7</td><td>support</td><td>13313</td><td>0.78</td></tr><tr><td>8</td><td>change</td><td>26627</td><td>0.90</td><td>8</td><td>law</td><td>11120</td><td>0.80</td></tr><tr><td>9</td><td>effect</td><td>23658</td><td>0.93</td><td>9</td><td>aid</td><td>5128</td><td>0.83</td></tr><tr><td>10</td><td>war</td><td>17868</td><td>0.90</td><td>10</td><td>help</td><td>3699</td><td>1.0</td></tr></tbody></table>

<figure><img src="image_6.png" /><figcaption>Fig. 5. The number of tweets per hour before and after the occurrence of the Osaka Northern Earthquake.</figcaption></figure>

All calculations result presented on the graph is showing the amount of usage data of each category during the Osaka North Earthquake on June 17-19, 2018. From this graph, the line of data in each category is following the same direction as the earthquake situation. However, the amount of content in each category is different (see Fig. 5).

As shown in the graph, the most tweet message on the graph is in the place category. Most of the content in place category refers to the location of the earthquake, such as “【地震情報】18日07時58分頃、大阪府北部で震度6弱の地震がありました。震源地は大阪府北部で、震源の深さは約10km、地震の規模はM5.9と推定されます。この地震による津波の心配はありません。この地震について、緊急地震速報を発表しています。震度
6弱:大阪北区、高槻市、枚方市など” (In English: [Earthquake Information] An earthquake of less than 6 intensity occurred in northern Osaka Prefecture around 7:58 a.m.on the 18th. The epicenter is in northern Osaka Prefecture, the depth of the epicenter is estimated to be about 10km and the scale of the earthquake is estimated to be M5.9. There is no concern about a tsunami caused by this earthquake. An earthquake early warning for this earthquake was announced. The seismic intensity of 6-lower: Kita Ward of Osaka, Takatsuki City, Hirakata City, etc.). Also, the second rank of the graph is damage categories because of the “intensity” keyword that has a volume of 2,409,063 tweets. The Tweet message contents about the strength of the earthquake and the shaking, for example, “07時58分頃、地震がありました。[震度6弱] 大阪北部 [震度5強] 京都南部 [震度5弱] 兵庫南東部、奈良 [震度4] 嶺南地方、滋賀北部、滋賀南部、大阪南部、淡路島 [震度3] 三重北部、三重中部、京都北部、兵庫北部、兵庫南西部、和歌山北部” (In English: An earthquake occurred around 07:58. [seismic intensity of 6-lower] northern Osaka [seismic intensity of 5-upper] southern Kyoto [seismic intensity of 5-lower] southeastern Hyogo, Nara [seismic intensity of 4] Reinan region, northern Shiga, southern Shiga, southern Osaka, Awaji Island [seismic intensity of 3] northern Mie, central part of Mie, Northern Kyoto, Northern Hyogo, Southwestern Hyogo, northern Wakayama.). Then the third of the overall graph will be the “transport” category related to travel. The most tweet content is about traveling by trains as they are the main transportation of Japanese people. The content is mainly about the train information and the train stopped disrupted information. It is helpful information. Also, the other two important information is data before and after an earthquake happens. We expand both graphs to see data fragmentation information (see **Error! Reference source not found.** and 7).

When we expand the graph, we can find the distribution of information. We found and the trend of the line graph in the group of transport categories that occurred before the Great Earthquake happens. There was a fewer tweet, but after the earthquake happens, it became the second most tweet. Another important graph is in the category Warning is the third highest in the pre-disaster period; however, after the disaster, the amount of tweets has decreased. The most content in that period of Warning category is “地震発生後余震への備え ①断水に備えお風呂に水を貯める ②停電に備え懐中電灯の用意 ③食器棚の扉にはガムテープで食器が落ちない工夫を…” (In English: Preparations for aftershocks: (1) Keep water in a bath to prepare for a water outage, (2) prepare flashlights for A power outage, (3) prevent falling objects, such as stopping the door of the cupboard with gum tape. ...). Also, the graph of Action and Service category that rises after the earthquake happens. In this section can be analyzed to find guidelines for helping when an earthquake happens.

However, the summary of the graph result is the amount of information for each category that cannot present the meaning in each message. There will be more than one keyword noun. Therefore, the next step is to count the frequency of the categories in each sentence. This step will help us know the meaning of the contents.
The sentence of the Twitter message is very short words (only 140-280 characters) and the keywords in each message use word's similarity scores as a comparison. Therefore, some words can be grouped in more than one category but have different scores. We have created calculation rules of counting as follows.

<figure><img src="image_7.png" /><figcaption>Fig. 6. The number of tweets per hour before the occurrence of the Osaka Northern Earthquake.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Fig. 7. The number of tweets per hour after the occurrence of the Osaka Northern Earthquake.</figcaption></figure>

* In the case that two categories with the same keyword, compare the similarity score of each category to find which category is more accurate.

* In the case that the keyword has more than one category, the second category that has top of similarity score will be used to find the frequency as well.

For example, in the case of the content of the Twitter that separates the nouns is [caution] [intensity] [prefecture] [preparation]. The keyword caution is a word in both categories Alert and Warning, but the score is different. The system retrieves both categories to counting frequency. Therefore, this content is in the group [Alert | Warning] [Damage] [Place] [Alert] and summarizes the results of the word frequency count as follows: Alert 2 = 2 / 5x100 = 40%, Warning 1 = 1 / 5x100 = 20%, Damage 1 = 1 / 5x100 = 20%, Place 1 = 1 / 5x100 = 20%. From the results of calculations, summary
the content of this Tweet message as a primary category "Alert" within that message, there has the content of category damage and category places inside.

We tested the accuracy of calculation frequency in content counting using the con- fusion matrix. We random 10,000 tweets from all tweets in the database to calculate the results of the accuracy. By comparing the result of prediction from the calculation pro- gram (from counting the frequency of words and classified into ten categories) and the actuality meaning of tweets when reading and translating typically by a human. (The results are shown in **Error! Reference source not found..**)

<table><caption>**Table 3.** Accuracy of classification.</caption><thead><tr><th></th><th>Accuracy</th><th>Precision</th><th>Recall</th></tr></thead><tbody><tr><td>confusion matrix score</td><td>0.874</td><td>0.97</td><td>0.861</td></tr></tbody></table>

From the results, it presents the accuracy score of the use of classifying messages in social media. The accuracy is as much as 87% because classified by use group of the category to reduce the variety of keyword in social media. Moreover, the use of word similarity effects with accuracy score is because the use of word similarity will select the words to analyze and group it into the category. These process decreases the number of keywords to find the frequency in each message.

We analyzed that the imperfection of accuracy is caused by a number of categories defined. We should increase the number of categories because the sentences have many kinds of words than categories that have been designed, such as question groups, etc. Also, other imperfection of accuracy is the length of sentences because the Twitter message can have characters only 140-280. It was not enough to find the max score of one category. As sentences must be two main categories, when looking at the actual content, then group into one category.

# 6 Conclusion and Future work

We proposed the method to classify tweets posted at the time of disaster. The verifica- tion experiment using Osaka North Earthquake tweet dataset showed that the accuracy of the proposed method is over 80%. In future research, it would be useful to compare the results obtained with other datasets of disasters and use different machine learning, such as SVM to compare the results to find out the midpoint of the score using for developing the best helping system for foreigners in the natural disaster situation in Japan.

We found that the accuracy of the classified content depends on the number of cate- gories defined. Therefore, it should increase the number of categories that affect the content of disaster to increase the accuracy of the result, especially the category that can separate contents between the question sentences and knowledge sentences. More- over, the system should have an information filter that correctly classifies fake infor- mation and truth knowledge information as well. However, we found some keywords that cannot be translated into English ultimately, such as the keyword "Neko" which
means “cat” in an English word. The problem has come from that the Twitter user does
not type correctly or type Japanese words in English, so this information was missing
some keyword. The database should have a table for translation Japanese important
proper nouns (including famous locations name and brands name) into English.

Acknowledgment

This study was supported by JSPS KAKENHI (Grant Number 18K11553) and the To-
kai University General Research Organization.

References

1. Kemavuthanon, K., Uchida, O.: Social Media Messages During Disasters in Japan: An Empirical Study of 2018 Osaka North Earthquake in Japan. In: Proc. 2019 IEEE 2nd International Conference on Information and Computer Technologies, 199–203 (2019).

2. Yamada, S., Utsu, K., Uchida, O.: An Analysis of Tweets during the 2018 Osaka North Earthquake in Japan -A Brief Report. In: Proc. 5th International Conference on Information and Communication Technologies for Disaster Management (2018).

3. Uchida, O., Rokuse, T., Tomita, M., Kajita, Y., Yamamoto, Y., Toriumi, F., Semaan, B., Robertson, S., Miller, M.: Classification and Mapping of Disaster Relevant Tweets for Providing Useful Information for Victims During Disasters. IIEEJ Transactions on Image Electronics and Visual Computing, 3(2), 224–232 (2015).

4. Uchida, O., Kosugi, M., Endo, G., Funayama, T., Utsu, K., Tajima, S., Tomita, M., Kajita, Y., Yamamoto, Y.: A Real-Time Information Sharing System to Support Self-, Mutual-, and Public-Help in the Aftermath of a Disaster Utilizing Twitter.IEICE Transactions on Fundamentals, E99-A(8), 1551–1554 (2016).

5. Layek, A. K., Pal, A., Saha, R., Mandal, S.: DETSApp: An App for Disaster Event Tweets Summarization using Images Posted on Twitter.In: Proc. 2018 5th International Conference on Emerging Applications of Information Technology (2018).

6. Sumalatha, M. R., Basta, P., Sinha, A., Shrinath, P.: Social Media for Disaster Relief —Geo Distributed Social Service System. In: Proc. 2015 7th International Conference on Advanced Computing (2015).

7. Mizuno, J., Tanaka, M., Ohtake, K., Oh, J.-H., Kloetzer, J., Hashimoto, C., Torisawa, K.: WISDOM X, DISAANA and D-SUMM: Large-scale NLP Systems for Analyzing Textual Big Data. In: Proc. 26th International Conference on Computational Linguistics: System Demonstrations, 263–267 (2016).

8. Naradhipa, A. R., Purwarianti, A.: Sentiment Classification for Indonesian Message in Social Media. In: Proc. 2012 International Conference on Cloud Computing and Social Networking (2012).

9. Cavalin, P. R., Moyano, L. G., Miranda, P. P.: A Multiple Classifier System for Classifying Life Events on Social Media. In: Proc. 2015 IEEE 15th International Conference on Data Mining Workshops, 1332–1335 (2015).

10. Li, H., Tian, Y., Ye, B., Cai, Q.: Comparison of Current Semantic Similarity Methods in WordNet. In: Proc. 2010 International Conference on Computer Application and System Modeling, V4-408-V4-411 (2010).
11. Nguyen, L. T., Huynh, K. M.: Using WordNet Similarity and translations to create synsets for Ontology-based Vietnamese WordNet. In: Proc. 2016 5th IIAI International Congress on Advanced Applied Informatics, 651–656 (2016).

12. Hasi, Nasun-Ur: The Automatic Construction Method of Mongolian Lexical Semantic Network Based on WordNet. In: Proc. 2012 Fifth International Conference on Intelligent Networks and Intelligent Systems, 220–223 (2012).

13. Pedersen, T., Patwardhan, S., Michelizzi, J.: WordNet::Similarity - Measuring the Relatedness of Concepts. In: Proc. Nineteenth National Conference on Artificial Intelligence (2004).

14. Zhang, Y.-L., Hasi: A Constructing Method of Mongolia-Chinese-English Multilingual Semantic Net based on WordNet. In: Proc. 2015 International Conference on Computer Science and Applications, 195–198 (2015).

15. Kale, S., Padmadas, V.: Sentiment Analysis of Tweets Using Semantic Analysis. In: 2017 International Conference on Computing, Communication, Control and Automation (2017).

16. Indra, S. T., Wikarsa, L., Turang, R.: Using Logistic Regression Method to Classify Tweets into the Selected Topics. In: Proc. 2016 International Conference on Advanced Computer Science and Information Systems, 385–390 (2016).

17. Janssens, O., Slembrouck, M., Verstockt, Hoecke, S. V., Walle, R. V.: Real-time Emotion Classification of Tweets. In: Proc. 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, 1430–1431 (2013).

18. Twitter. https://twitter.com/

19. Riyadh, A. Z., Alvi, N., Talukder, K. H.: Exploring Human Emotion via Twitter. In: Proc. 2017 20th International Conference of Computer and Information Technology, 22–24 (2017).

20. Matsumura, N., Miura, A., Komori, M., Hiraishi, K.: Media and Sentiments in the Great East Japan Earthquake Related Tweets – Social Media as “Meta Media”. In: Proc. 2016 IEEE Tenth International Conference on Semantic Computing, 465–470 (2016).

21. Google API, https://cloud.google.com/translate/