Slang feature extraction by analysing topic change on social media

CAAI Transactions on Intelligence TechnologyVolume 4, Issue 1 p. 64-71 Research ArticleOpen Access Slang feature extraction by analysing topic change on social media Kazuyuki Matsumoto, Corresponding Author Kazuyuki Matsumoto matumoto@is.tokushima-u.ac.jp orcid.org/0000-0002-9820-1470 Graduate School of Technology, Industrial and Social Sciences, Tokushima University, 2-1, 770-8506 Tokushima-shi, Minamijosanjima-cho, JapanSearch for more papers by this authorFuji Ren, Fuji Ren Graduate School of Technology, Industrial and Social Sciences, Tokushima University, 2-1, 770-8506 Tokushima-shi, Minamijosanjima-cho, JapanSearch for more papers by this authorMasaya Matsuoka, Masaya Matsuoka Graduate School of Technology, Industrial and Social Sciences, Tokushima University, 2-1, 770-8506 Tokushima-shi, Minamijosanjima-cho, JapanSearch for more papers by this authorMinoru Yoshida, Minoru Yoshida Graduate School of Technology, Industrial and Social Sciences, Tokushima University, 2-1, 770-8506 Tokushima-shi, Minamijosanjima-cho, JapanSearch for more papers by this authorKenji Kita, Kenji Kita Graduate School of Technology, Industrial and Social Sciences, Tokushima University, 2-1, 770-8506 Tokushima-shi, Minamijosanjima-cho, JapanSearch for more papers by this author Kazuyuki Matsumoto, Corresponding Author Kazuyuki Matsumoto matumoto@is.tokushima-u.ac.jp orcid.org/0000-0002-9820-1470 Graduate School of Technology, Industrial and Social Sciences, Tokushima University, 2-1, 770-8506 Tokushima-shi, Minamijosanjima-cho, JapanSearch for more papers by this authorFuji Ren, Fuji Ren Graduate School of Technology, Industrial and Social Sciences, Tokushima University, 2-1, 770-8506 Tokushima-shi, Minamijosanjima-cho, JapanSearch for more papers by this authorMasaya Matsuoka, Masaya Matsuoka Graduate School of Technology, Industrial and Social Sciences, Tokushima University, 2-1, 770-8506 Tokushima-shi, Minamijosanjima-cho, JapanSearch for more papers by this authorMinoru Yoshida, Minoru Yoshida Graduate School of Technology, Industrial and Social Sciences, Tokushima University, 2-1, 770-8506 Tokushima-shi, Minamijosanjima-cho, JapanSearch for more papers by this authorKenji Kita, Kenji Kita Graduate School of Technology, Industrial and Social Sciences, Tokushima University, 2-1, 770-8506 Tokushima-shi, Minamijosanjima-cho, JapanSearch for more papers by this author First published: 04 February 2019 https://doi.org/10.1049/trit.2018.1060Citations: 7AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Recently, the authors often see words such as youth slang, neologism and Internet slang on social networking sites (SNSs) that are not registered on dictionaries. Since the documents posted to SNSs include a lot of fresh information, they are thought to be useful for collecting information. It is important to analyse these words (hereinafter referred to as 'slang') and capture their features for the improvement of the accuracy of automatic information collection. This study aims to analyse what features can be observed in slang by focusing on the topic. They construct topic models from document groups including target slang on Twitter by latent Dirichlet allocation. With the models, they chronologically the analyse change of topics during a certain period of time to find out the difference in the features between slang and general words. Then, they propose a slang classification method based on the change of features. 1 Introduction With the expansion of social networking sites (SNSs) such as Twitter, the Internet has penetrated into daily life. The documents posted on SNSs by many users should be useful for collecting timely information. However, the documents on SNSs often include youth slang, neologism and Internet slang that are not registered into dictionaries. Therefore, analysis of slang is important for allowing extraction of new information with high accuracy. This study focuses on the knowledge that the usage of some slang words varies over time. Therefore, we analyse a set of tweets including slang with latent Dirichlet allocation (LDA) and output their topics monthly. Then, the randomly selected topics are set as a basis, and the similarities between the topics are calculated for each month. We investigate the topics where the similarities are greatly changed from the basis so that we can examine the characteristics of the slang based on the change of topics. We also propose a method to judge the words, whether they are used as slang, from the obtained topic change features. This paper comprises six sections. Section 2 describes the related works of slang analysis or slang processing. Section 3 explains our proposed method to analyse slang features. In Section 4, the experiments and results are illustrated. Section 5 discusses the analysis result and the problems of the proposed method. Finally, Section 6 concludes the paper. 2 Related works There are previous studies focused on topic detection, topic tracking or detection of slang. As follows, the related works are explained in sequence. 2.1 Topic detection Recently, latent topic analysis has focused on the field of text/web mining or natural language processing. LDA [[1]] is a famous algorithm for topic modelling. Studies on what kinds of topics appear in a document are included in the study field of topic detection. There are studies for electronic text data, such as web data [[2]–[13]]. Although there are various types of documents such as news articles, e-mails, weblogs, among others, document classification topics are often annotated with correct labels. Kimura et al. [[14]] proposed the classification method of the relation between hashtags by considering co-occurrence and latent topic from tweets. Their method can identify the meaning of unknown hashtag by classifying relation with known hashtags. They defined the relation of hashtags as four types. As an evaluation experiment, the proposed method based on co-occurrence and the latent topic could achieve higher accuracy than the baseline methods which use only co-occurrence or latent topic. However, there might be more than one topic for each document, and unexpected topics actually often appear. Therefore, LDA has been used more recently as a method that does not define topics in advance. Hashimoto et al. [[15]] proposed a topic detection method by using a neural network-based vector space model to calculate semantic similarities between system review documents. Their method can achieve a higher sensitivity of eligible studies and a reduced manual annotation cost than the baseline method using the LDA model using MALLET toolkit [[16]]. Ren et al. [[17]] focused on predicting users' opinions toward specific topics they had not directly given yet. They proposed the method based on a social context and topical context incorporated matrix factorisation framework. 2.2 Topic tracking There are some studies that do not detect topics but track the topics down in chronological order. This section introduces the studies about topic tracking methods with latent topics. Serizawa et al. [[18]] use LDA to extract topics and then judge and integrate the similarities between the topics to decide the appropriate number of topics. Based on the topic number, they propose a method to extract/track-down the topics. In this study, we appropriately determine the topic change that can be found from the change of similarities of each month and analyse the features of the topics, including slang from the tendency of a topic change. Zhao et al. [[19]] proposed a dynamic query expansion (DQE) model for theme tracking in Twitter. Their proposed model can express the theme consistency among heterogeneous entities through semantic and social relationships. They conducted the experiment for tracking the theme 'civil unrest.' As a result of that, they could confirm the effectiveness and scalability of DQE. 2.3 Neural topic modelling Wan et al. [[20]] proposed a model that combined a deep neural network with a latent topic model. Cao et al. [[21]] proposed a neural topic model and its supervised extension. They evaluated the performance of their neural topic models on the document-oriented supervised tasks including multi-class classification, multi-label classification and regression. As a result, their proposed method achieved higher accuracy than the other supervised topic modelling methods. Larochelle et al. [[22]] proposed a model to learn semantic representations of texts from an unclassified text corpus. The evaluation experiments proved that their proposed model gives a good performance as a generative model of text documents and as a document representation learning algorithm. 2.4 Slang analysis This section introduces several studies focusing on the analysis of unknown expressions, such as slang. Hisano et al. [[23]] judge the similarity of proper nouns by using time-series correlation and word co-occurrence rate and then construct a similar word dictionary. With this dictionary, slangs that are quasi-words of proper nouns and abbreviations of proper nouns can become manageable. The study by Hisano et al. extracts similar words by targeting a specific topic. However, we would like to deal with various kinds of slangs that are not only words expressing specific things, but also slang expressing general events such as 'posharu' (fizzle out) and words having both meanings as slang and as general words, such as 'chiiter' (cheater) and 'teppan' (a sure thing). Our study also differs by focusing on semantic elements such as usage of slang. Matsumoto et al. [[24]] proposed the topic analysis based on latent topic modelling and word distributed representation. Their method analysed the change of topic or meaning of slang by using the topic keyword distributed vector and word distributed representation vector, which are calculated from the large tweet text corpus. As a result of the evaluation experiment, it was found that the change of slang's topic or meaning can be observed by using sentiment analysis and similarity calculation of topic modelling or word distributed representation. Their method focused on the topic change of positive and negative. However, the sentiment or impression of slang is often changed in a short period. So, we focus only on fluctuation of topic information of slang in a short period. 2.5 Sentiment and semantic analysis on big data Ren et al. [[25]–[27]] proposed the sentiment and semantic analysis method on big data on the Internet. On the other hand, Matsumoto et al. [[28]] proposed the personality predicting method from users' tweets based on neural networks and sentence embedding. Sentiment analysis of web text or personality analysis of short text plays a very important role in many text mining tasks. Topic modelling should also be an effective and important component of sentiment analysis or personality analysis. Ren et al. [[29]] proposed the emotion analysis method on social big data. Their method can obtain time-series emotion fluctuation for each Twitter users' cluster. They used word embedding method to extract text feature from short text on Twitter. A lot of short text data include noise, such as meaningless expressions or slangs. Many big data include such expression, so, we think that topic modelling is effective to extract meaning or sentiment indirectly. 3 Proposed method This section describes the proposed method. 3.1 Equations This section describes the procedure of similarity calculation between topics in each subsection, which is used in this study to investigate the change of topics. By tracking the fluctuation of topics, we analyse what meaning was used for each slang term. Fig. 1 shows an example of calculating topic similarity and output of higher ranked topics regarding the word 'sweets' in April and May. The process of topic analysis is explained in order below. Fig. 1Open in figure viewerPowerPoint Calculation of topic similarity between April and May 3.1.1 Extraction of word set First, the target months and slang are decided in the database, and then the wordlist is obtained from the target tuple. Next, the word set is made by extracting only nouns, unknown words, and slang from the wordlist. In this process, the tweets that produced the completely matched word sets are regarded as the mechanically posted tweets and removed, leaving only the first-appeared tweets as a target for analysis. 3.1.2 Topic generation Topics are generated by inputting the data of the reshaped word set. At this time, the hyper parameters are set as α = 0.5, and β = 0.1, the number of words for each topic is set as 10, and the number of topics is set as 10. Here, we explain how we decide the number of topics. As for the 40 kinds of the targeted Japanese slangs, we, respectively, prepare a corpus consisting of the tweets including each Japanese slang. We calculate the LDA model perplexity (1) for each number of topics from 3, 4, 5, 6, 7, 8, 9 to 10. The LDA model perplexity indicates the performance of model prediction and is defined by the inverse of probability. We use the number of topics that obtained the lowest averaged value of perplexity. In this study, we set the number of topics as 10. To decide the parameter values of α and β, we calculated the average number of topics for each tweet and selected the combination of the parameters when the least number of topics was obtained. As this result, we set α = 0.5 and β = 0.1 (1) where M is the number of documents, d is the document, wd, is the words appeared in document d, Nd is the number of words in document d, and p (wd) is the appearance probability of word wd. The example result of the output is shown in Table 1. The result shows the word set of the slang 'sweets' in topic 2 in April. Table 1. Slang 'sweets' of topic 2 in April Feature word Appearance probability laugh 0.0778 sweets 0.0742 girl 0.0438 series 0.0379 boy 0.0297 I 0.0156 thing 0.0108 woman 0.0103 previous 0.0101 it 0.0093 3.1.3 Similarity calculation The topics are chosen to be used for analysis basis. Then, the similarities with the topics of other months are calculated, and the top 3 similarities are output. The values are calculated for each month so that the transitions of the top 3 similarities are analysed monthly. When we calculate similarities, the vector consisting of the feature words and their appearance probabilities are used. However, because the tweets including the target slang are targeted for analysis, there is a problem in that the appearance probability of slang itself becomes very high. Generally, tf-idf is often used as a solution to this problem. Tf-idf is a value that multiplies term frequency by inverse document frequency. In the topic model, the tf-idf value can be implemented by using proposed by Blei et al. [[30]]. The calculation formula of is shown by (2) where is the appearance probability of v in topic k and K is the number of topics (k ∈{1,..., K }).We weight the term appearance probability using . After that process, we decide the corresponding to the feature words as the vector x in that topic and calculate cosine similarities between the topics with (3). The calculated cosine similarities are used for analysis as the similarities between the topics (3) 3.1.4 Latent Dirichlet allocation To analyse the usage of slang, we focus on the analysis of the latent topics. In this study, we analyse the latent topics by using LDA. LDA is a probabilistic topic model that assumes multiple topics exist in a document. The graphical model of the LDA and the explanation of each variable are shown in Fig. 2 and Table 2. Table 2. Parameters of LDA α topic occurrence parameter β word occurrence parameter of topic K K number of topics M number of documents N number of words in document M θ topic distribution in document M Z topic of word n in document M W word Fig. 2Open in figure viewerPowerPoint LDA graphical model The steps of document generation by LDA is as follows. Dir indicates Dirichlet distribution, and Multinomial indicates multinomial distribution. In this study, we calculate LDA by using the tool: GibbsLDA++ [[31]]. The distribution of topic –Dir(α) is generated for each document i ∈{1,..., M }. The distribution of word –Dir(β) is generated for each topic k ∈{1,..., K }. The topics –multinomial () are generated for i ∈{1,..., M } and each word j ∈{1,..., Ni }. Words –multinomial () are generated. 3.1.5 Database This study targets the tweet sentences including slang for analysis. Since we would like to focus on the time-series analysis, tweet sentences need to be collected for each month. Therefore, we construct a database to more effectively use the collected documents. Table 3 shows the structure of each table. It consists of four columns: tweet posting date (year, month, day, hour, minute, second), the account ID of the user who posted the tweet, tweet sentences, and the data with format of <Word/POS> obtained by morphologically analysing tweet sentences with MeCab [[32]]. Table 3. Structure of the table in the slang tweet database Field name time nameID Tweet wordlist data type date time text text text At this time, the 654 slang words are already registered in a user dictionary in MeCab and the '-unk-feature UNK' is set as an option. Currently, we use SQLite3 [[33]] as the database-management system, and 148,551,674 tweets are registered into the database. We would like to update the database in the future. 3.2 Topic fluctuation analysis This subsection describes the method to calculate scores for each topic by month. In this process, we use the same database, wordlist, and LDA as those that are used in the previous section. Fig. 3 shows the system flow. Fig. 3Open in figure viewerPowerPoint System flow (topic fluctuation analysis) 3.2.1 Extraction of word set We select the slang from the database and obtain the wordlist from the target tuple. Next, we make the word set by extracting only nouns, unknown words, and slang from the wordlist. During this process, we remove the tweets that produced the word sets completely matched to the other tweets' word sets from the analysis target. As the stop words, we remove the words that do not include kanji, hiragana, or katakana characters because these words are not Japanese. 3.2.2 Topic generation We generate topics by using all wordlists of each month as input data. During this process, we set the hyper parameters as α = 0.5, β = 0.1, and the number of words for each topic is set as 100 and the number of topics is set as 100. 3.2.3 Topic clustering If the number of topics is set as 100, a lot of similar topics will be generated. Therefore, after the analysis by LDA, we reduce the number of topics by using the unsupervised clustering method. After the clustering, we use the centroid vector of each cluster as the representative vector of each topic. In this step, the number of clusters is automatically decided by setting the threshold of similarities between vectors. 3.2.4 Topic scoring We obtain the wordlist for each month and annotate the score for each topic. The equation of topic scoring is shown by (4) where is the probability of occurrence v of word i in topic k; is the number of tweets in month m; is the number of tweets in which word i appeared in month m; and is the feature word in the topic k. At the time of scoring the tweets, if the feature words appeared in the tweet sentences, we add the corresponding appearance probability as a feature quantity. Finally, the averaged value calculated by the number of tweets is defined as the score of the topic. Since the score can be represented by multiplying the appearance probability of the target word and the number of tweets including the target words, the calculation is the same as (3). In Experiment 2, we annotate the score for each topic by using this scoring method and output the monthly fluctuation. 3.3 Comparison with tweet distributed representation One of the problems regarding the analysis on tweet topic change by the topic vector is that the relation of word co-occurrence cannot be obtained sufficiently because a tweet (document), which is targeted for analysis, is short. Besides, in simple bag-of-words model, weights such as tf-idf are calculated on words to measure word importance value. However, this causes too high importance values on rare expressions appearing in only specific topics. On the other hand, there is a study reporting that the word distributed representation and the document representation learning are more effective to extract short text feature. Liu et al. [[34]] proposed the task-oriented word embedding for the text classification method. From the results of evaluation experiments, it was found that their approach significantly outperformed the state-of-the-art methods. Boom et al. [[35]] found that high performance is shown in the short text representation learning task by weighting word distributed representation with idf value. As in their study, it is effective to learn text representation based on word distributed representation. In this section, we create a model that generates a fixed dimensional real-valued vector by training distributed representations of tweets. We extract a 64-dimensional vector from tweets based on this model. We generate a cluster vector for each month with consideration of cluster appearance frequency and similarity, by conducting clustering on the extracted vector set. We assume that this vector should express the meaning on the tweet set. We calculate the similarity between the cluster vectors obtained from the tweet set of the neighbouring month and detect the semantic change of the slang by finding out when the similarity becomes lower than the average similarity. We also use topic vector similarity to analyse the tendency appeared. Since the number of tweets is different in each month, we limit the number of tweets up to 100 and create a distributed representation cluster vector by randomly extracting the 100 tweets from each month. To learn the tweet distributed representations, we use pre-trained word distributed representation as a feature in this study. We use the nouns extracted from each tweet as the training data, and trained 64-dimensional distributed representations by word2vec [[36]] using the skipgram algorithm and setting context window size as 5. We randomly extracted one million tweets as target tweets. Then, the words in each tweet are replaced by the 64-dimensional distributed representation vector. For each tweet, vector up to 32 words are input, and the encoder–decoder based on convolutional neural networks is learned. From the hidden layer of the learned encoder–decoder, the 64-dimensional vector responding to the inputted tweet. This network model is hereinafter called as CNN-AE. Fig. 4 shows the layer construction of CNN-AE. Fig. 4Open in figure viewerPowerPoint CNN-AE: auto encoder–decoder We used ReLU as an activation function between the hidden layers and Softmax as an activation function for the output layer. As the optimisation algorithm, we used Adam and set the mini batch size as 256 and learning epochs as 1000. The output of the fifth layer (hidden layer) is extracted as the 64-dimensional vector. We conduct clustering on the tweets with the k -means method by using the trained model. The number of clusters was set as 60, 100, 200, 300, 400 and 500. We assume that the target slang should have its own tendency to appear in each independent topic. Therefore, if several different slangs co-occurred in the same tweet in the same cluster, we can regard that classification by topic should have been failed. We judge the result of the clustering result according to the average value of the co-occurrence frequency between the different slangs in the same cluster. When the value is smaller, we judge it as better clustering result. As a result of the clustering, the optimal number of clusters was N = 200. Therefore, in this study, we set N = 200 and analyse the semantic change of slangs by using the feature vector (semantic feature vector), which is created based on the appearance frequency in each month of the tweets belonging to each cluster instead of a topic vector. 4 Experiments This section describes the experiment of the proposed method. 4.1 Analysis of topic change: experiment 1 We investigated the transition of each topic score in the term from April 2015 to December 2015 by using the method described in the previous section. We randomly selected a basis topic from the topics in April and calculated similarities with the topics from May 2015 to September 2015. In this study, we target 40 kinds of Japanese slangs shown in Table 4. Table 4. Part of the target Japanese slangs boomerang oni (devil) gobaku (bombing error) enban (disk) yuri (lily) Jizou (a stone statue of Jizo) kami (god) teppan (grill) atsui (hot) pizza urayama (a mountain at the back) daishouri (big winning) Gunnmar (people of Gunnma prefecture) daisougen (prairie) anntei (stabilisation) sweets kuudere jiwaru Youtsube (YouTube) denntotsu The topic similarity was calculated by using cosine similarity between the vectors of topics, of which dimension is word and value is occurrence probability. As a result, we found the tendency of some words. The examples are described in the following subsection. 4.1.1 Topic occurrence: 'boomerang' Fig. 5 shows the score transition which was obtained by analysing the slang word 'boomerang.' A graph is displayed with an analysis result visualisation tool which was created by a graph plot library of JavaScript called Chart.js [[37]]. Fig. 5Open in figure viewerPowerPoint Similarity fluctuation in topic 0 of slang 'boomerang' From this figure, we can see that the similarities are increasing from June to July. Therefore, we focused on the topics that calculated the highest similarity from June to September. Table 5 shows the feature words of each topic. Table 5. Similar topics of slang 'boomerang' April, topic 0 as a basis Apr. (T0) Jun. (T3) Jul. (T2) Aug. (T7) Sep. (T6) boomerang boomerang boomerang thing boomerang myself laugh thing people thing I partner laugh boomerang myself yo weapon myself myself yo what head I minshuto people video myself flaming yo child statement up statement what I all what kuse statement face flaming change mon congressmen outside san I number criticism story From this table, 'boomerang' in topic 0 in April means that the calumny or criticism towards other people goes for the persons themselves who posted such words. This indicates that the word 'boomerang' is used as slang in a so-called 'boomerang statement.' In fact, the word 'boomerang' co-occurs with the word 'statement' in the topic. Next, we focus on the topics from June and July. In June, 'boomerang' was not used as slang in meaning. However, in July, 'boomerang' was found to be used as slang. On the other hand, 'boomerang' as slang appeared in the topic of August. However, in the topic of September when the similarity decreased, the latent topic of 'boomerang' cannot be judged by a human. As a result, it was found that the topic fluctuation occurred by topic similarity change. Moreover, the topic keywords are thought to occur among the topic basis when the similarity increases. The topic keywords are thought to fall into decline when the similarity decreases. 4.1.2 Topic continuance: 'sweets' Next, Fig. 6 shows the transition of similarity when the topic 2 of the slang word 'sweets' was set as a basis. Fig. 6Open in figure viewerPowerPoint Similarity transition of slang 'sweets' in topic 2 From this figure, it is found that the similarities repeatedly increased and decreased. By focusing on the topics which obtained the highest similarity in the period from June to September, the feature words for each topic are shown in Table 6. Table 6. Similar topics with topic 2 of slang 'sweets' in April Apr. (T2) Jun. (T9) Jul. (T5) Aug. (T1) Sep. (T6) laugh laugh laugh laugh sweets sweets sweets sweets sweets laugh girl girl girl girl girl series like series series people boy thing 'symbol' boy thing I people boy people like thing focus another stomach kun focus woman woman power story power previous myself stomach man I it yo yeah like sweets (laugh) The word 'sweets' as slang has two meanings: (i) the sweet dessert and (ii) an ironic meaning regarding people who regurgitate information by mass medium or bandwagon in the world. From this table, 'laugh', 'sweets', and 'girl' are ranked higher in every month so that the latent topic can be confirmed as the slang 'sweets' having ironic meaning. 4.2 Analysis of topic score transition: experiment 2 4.2.1 Topic score transition: 'boomerang' The score transition obtained by analysing slang 'boomerang' is shown in Fig. 7. We found that the highest score was calculated in May. Fig. 7Open in figure viewerPowerPoint Score transition of slang 'boomerang' It is also found that the score of topic 3 increased until Decem