Best feature performance in codeswitched hate speech texts

best feature performance in codeswitched hate speech texts abstract How well can hate speech concept be abstracted in order to inform automatic classification in codeswitched texts by machine learning classifiers ? We explore different representations and empirically evaluate their predictiveness using both conventional and deep learning algorithms in identifying hate speech in a ~48k human-annotated dataset that contain mixed languages , a phenomenon common among multilingual speakers . This paper espouses a novel approach to handle this challenge by introducing a hierarchical approach that employs Latent Dirichlet Allocation to generate topic models that feed into another high-level feature set that we acronym PDC . PDC groups similar meaning words in word families during the preprocessing stage for supervised learning models . The high-level PDC features generated are based on Ombui et al , ( 2019 ) hate speech annotation framework that is informed by the triangular theory of hate ( Stanberg,2003 ) . Results obtained from frequency-based models using the PDC feature on the annotated dataset of ~48k short messages comprising of tweets generated during the 2012 and 2017 Kenyan presidential elections indicate an improvement on classification accuracy in identifying hate speech as compared to the baseline Keywords : Hate Speech , Code-switching , feature selection , representation learning I. introduction Identifying hate speech in short text messages retrieved from social media is a challenging classification task . This data is often unstructured and noisy , coupled with the different dimensions of big data which include massive volume , high-velocity , questionable veracity , complex variety , and impactful to society ( Burnap & Williams , 2015 ) . The negative ripples of hate speech on social media , easily amplified across time , geographical and legal jurisdictions , not only abrogate user experience but have also been shown to turn into real-life violence in some cases ( Hatzipanagos , 2018 ) . This phenomenon is closely associated with online harassment , cyberbullying , body shaming and other hateful attacks targeting individuals or groups on the basis of belonging to a protected characteristic like race , ethnicity , religion , gender , etc . Social media companies are increasingly under pressure by various stakeholders to better respond to the phenomenon . Evidently , all social media networks have a user policy on hate speech content on their platforms . However , most of these companies mostly rely on users to flag such content which then goes through some element of manual review to ascertain whether it violets the user policy on hate speech content ( Sandler , 2018 ) . Subsequently , a decision is made to either to block the content , terminate the account , or take some other predefined action . This approach apparently is unfeasible in the light of big data . The research community has responded well over time , clearly by the growing number of research activities including workshops , conferences , software applications , and publications in the area of automatic hate speech identification ( Warner and Hirschberg,2012 ; Kwok and Wang , 2013 ; Gitari et al. , 2015 ) , offensive language detection ( Xu and Zhu , 2010 ; ) and other related areas like cyberbullying ( Cynthia et al.,2015 ) , fake news detection , sentiment analysis , and stance detection . However , most of the studies have focused on European languages , especially English , whereas there are so many other languages used on social media platforms , especially regional and resource-scarce languages like Swahili which is a lingua franca in East and part of central Africa . Besides , monolinguals form about 40 % of the world population , whereas bilinguals , multilinguals , and polyglots form about 60 % . ( Ansaldo et al. , 2018 ) We postulate that this statistics is increasingly being mirrored on social media with the evidence of codeswitched language in datasets , for example the dataset that was built in our study that comprises of ~400k tweets collected during the 2017 presidential campaigns in Kenya . Apparently , majority of Kenyans are multilingual , speaking their mother tongue ( L1 ) , Swahili as the national language ( L2 ) , and English as the official language ( L2 ) ( Muaka , 2011 ) . Therefore , this study bridges the gap for codeswitched language datasets in regards to automatic hate speech identification . To the best of our knowledge , this is the first study to collect and build a classifier for a codeswitched language dataset , specifically in English , Swahili , Sheng ( slang ) and some instances of words from native languages like Gikuyu and Luo . An example text message is , “ Hawa waarabu wanatorture kenyans wadhani sisi ni madogy zao ..... Being our employers does n't mean u own us . ” [ Translation : “ These Arabs are torturing Kenyans thinking we are their dogs… . Being our employer does not mean you own us. ” ] for hate speech identification in a codeswitched language dataset . Both , conventional and deep learning features and learning algorithms were explored on a ~48k human-annotated dataset using supervised machine learning approach . We actualize the PDC high-level features by mapping them into tf-idf vector representations for the task of hate speech identification . Several experiments using the PDC features prove their effectiveness in capturing hate in short text messages , with Passion ( P ) and Distance ( D ) components being the most salient with accuracies of 74.3 % and 74.2 % respectively . The PDC combined feature set is compared to the conventional features and the human annotation for the same dataset as baseline . II . related work There are several studies that have been conducted in relation to automatic hate speech detection . These can be classified into two main groups : those dealing with general hate speech as a binary classification task and those as a multi-class task . There are more studies in literature dealing with the phenomenon as a binary classification problem whereby the focus is on identifying a specific type of hate speech like racism ( Lozano et.al , 2017 , Tulkens et al. , 2016 ; Kwok & Wang , 2013 ) , anti-Semitism ( Warner & Hirschberg , 2012 ) or neither . For the second class , the argument is that there is need to differentiate the different types of hate speech or hate speech from offensive messages because conflating them does not match the ground truth . ( Davidson et al. , 2017 ; Badjatiya et al. , 2017 ; Fortuna , 2017 ) . There are various features that have been used previously in the detection of hate speech in social media text messages including demographic information of the users like the gender , location , ( Waseem and Hovy,2016 ) , syntactic features like the length of the message , part-of-speech tags as features and other high-level features , with dictionary lists and part-of-speech tags being popularly used . For the low-level features , the popular feature among the conventional learning algorithms has been the use of word frequency features like Bag-of-Words ( BoWs ) , n-grams ( both at character and word-level ) and Term Frequency-Inverse Document Frequency ( tf-idf ) features . Word embeddings as features for deep learning neural network algorithms have increasingly become popular in hate speech detection studies ( Badjatiya et al. , 2017 ; Hasanuzzaman et al. , 2017 ; Djuric et al. , 2015 ) . The popularly used pre-trained embeddings include Global Vectors ( GloVe ) , FastText n-grams , and Word2Vec text representations , at both character , word and sentence levels . The general feature usage frequency in automatic hate speech studies is well summarized in figure 1. figure 1 : Feature usage frequency in literature Some key observations from studies in literature include the popularity of conventional machine learning algorithm in learning classifiers and that English has been the most popular language of the datasets . According to available literature there very few in other languages like Dutch ( Tulkens et al. , 2016 ) , Amharic ( Mossie & Wang , 2018 ) , Arabic ( Al-Hassan & Al-Dosari , 2019 ) and none in Swahili . In addition , text messages ( tweets ) from Twitter have been popular with most of these studies . Besides , racism has been the most published type of hate speech . However , hardly any of the studies have publicly shared their full datasets apart from Davidson et al ( 2017 ) . III . method The general methodology that was used to build the classifier models for hate speech classification in short text messages in this study was based on the CRoss-Industry Standard Processes for Data Mining ( CRISPDM ) . Therefore , this study ’ s pipeline entailed problem understanding , data acquisition and preparation , feature engineering , modelling , and evaluation . 0 2 4 6 8 10 12 B O W B ro w n c lu st e rs C o n te xt u al … D ic ti o n ar ie s Eb in ar y co u n t… H as h ta gs Le n gt h M en ti o n s n -G ra m s N u m o f… O th e ri n g P ar ag ra p h 2 V ec P at te rn m at ch in g P O S P u n ct u at io n s R et w ee ts R u le -b as e d … Se n ti m e n t… Te m p la te -… Te xt f e at u re s TF -I D F To n e an al ys is To p ic m o d e lli n g To p ic s im ila ri ty Ty p ed … U R Ls U se r fe at u re s W o rd 2 V e c W o rd se n se … Feature Frequency conducted whereby several related studies and hate theories were examined . In addition , a content analysis was done on various definitions of hate speech as found on user policy guidelines of leading social media networks , and legal definitions of hate speech on constitution documents of various countries , with a focus on the case of Kenya . This was further informed by the need of automating the monitoring for hate speech on social media by key government agencies in Kenya like the National Cohesion and Integration Commision , which is in charge of matters related to hate speech ( NCIC , 2019 ) , the Kenya Education Network that is the primary Internet Service Provider of all tertiary learning institutions in the Country ( Kenet , 2018 ) , and other national security agencies . From this phase , a working definition of hate speech was derived , based on the section 13 of the NCIC Act of 2008 i.e . Hate speech is any message that incites hatred towards a target , whether a person or a group of people , on the basis of belonging to a protected group characteristic like ethnicity , race , religion , etc . A. data . There were 397,555 raw text messages collected . These mainly consisted of tweets from the 2017 general election in Kenya , including the repeat presidential elections held later the same year . Additional tweets from the 2012 general elections were also scrapped to form a formidable raw corpus size . This specific period was informed by existing research which indicates that there is often a higher volume of tweets generated during trigger events and not much thereafter ( King & Sutton , 2013 ) . Unlike text messages from other social media networks , tweets were purposively chosen because they are often topically structured , publicly available , and programmatically accessible via Twitter APIs , python tweet collection libraries , and even using custom-built crawlers . Moreover , Twitter features that enable users to anonymously engage in public discourse over topical issues in real time makes it a conducive platform for hate speech propagation . Besides , several previous studies in automatic hate speech detection have used tweets ( Kwok & Wang , 2013 ) ( Burnap & Williams , 2015 ) ( Waseem & Hovy , 2016 ) . The main data acquisition strategy employed was bootstrapping , starting with a seed of known hateful keywords ( Waseem , 2016 ) , phrase patterns ( William Warner & Hirschberg , 2012 ) , then followed by apparent hateful hashtags during the presidential elections period . In addition , tweets from pro- hate speech Twitter user accounts ( Kwok & Wang , 2013 ) , especially bloggers and politicians who often post content bordering hate speech were collected . Subsequently , this raw dataset was processed and culminated with a clean dataset of kkkk tweets . The data cleaning process included lowercasing , the removal of tweets with unprintable characters , that is the first thirtytwo character in the 7-bit ASCII code , removal of non-alphanumeric and duplicate tweets , retweets , emojis , and advertisements that often ride on trending hashtags . Examples of tweets with unprintable characters include “ Ã¶ ë¨¼ì¼ì´ì•¼ ” , and with advertisements include “ # NoReformsNoElections Apple launch iPhone X their most expensive phoneâ€¦ ” . Moreover , actual mentions were replaced with “ USERNAME ” tag , and hashtag symbols replaced with “ HASHTAG ” . In addition , tweet character length was used . For example below tweets with a length of less than 7 characters were removed . These comprised of tweets with only one or two-characters , with less than 3 word replies , which apparently did not meet the annotation scheme ’ s threshold for the predefined classes of this research . Examples of these include “ c ” , “ ok ” , “ DAAMN ! ” , “ I will ” . For other very short messages , they could not be contextualized because the referenced message was not available or they comprised of only numbers , one-word acronym , or a symbol . Examples of these include “ 2546 ” , ” WTF ” , ” Smh .. nkt ! ” , “ # ” . Although some words , whether unigram or bigrams or trigrams , were offensive , they did not have an identifiable target of hate and therefore got dropped too using the criteria of ‘ less than 4 ’ word length . For example , “ F # ck you all ! ” . In regard to tweet length , the conventional tweet message length has been 140 characters , which now has been doubled to 280 characters . In the contrary , from the observations made in this study , the longest tweet had 991 characters with two long URLs as part of the tweet message . This exception is explained by twitter ’ s design that puts a limit of 280 characters only on the message , whereas anything else thereafter e.g . the URL address can be of any length . Given that the length of a message could probably be an informative classification feature ( Davidson et al. , 2017 ) for the study , only the message length was considered , and therefore all the URL part of the tweets were dropped . 0 5000 10000 15000 1 2 5 4 9 7 3 9 7 1 2 1 1 4 5 1 6 9 1 9 3 2 1 7 2 4 1 2 6 5 2 8 9 3 1 3 3 3 7 3 6 1 Message Length Series1 Series2

Summary: This paper compared several classification methods, including deep neural networks (DNN), to identify hate speech texts. Mainly the data was corrected from twitter. The data was prepossessed to deal with by popular classification methods. The LDA and PDA are used to construct the identifier with high accuracy. Finally, Practical data was used to assess the performance of several machine learning algorithms.