Best feature performance in codeswitched hate speech texts
Authors: Edward Ombui
Lawrence Muchemi
Peter Wagacha
Venue:  (2020)
Keywords: Hate Speech
Code-switching
feature selection
representation learning

Abstract: How well can hate speech concept be abstracted in order to inform automatic classification in codeswitched texts by machine learning classifiers? We explore different representations and empirically evaluate their predictiveness using both conventional and deep learning algorithms in identifying hate speech in a ~48k human-annotated dataset that contain mixed languages, a phenomenon common among multilingual speakers. This paper espouses a novel approach to handle this challenge by introducing a hierarchical approach that employs Latent Dirichlet Allocation to generate topic models that feed into another high-level feature set that we acronym PDC. PDC groups similar meaning words in word families during the preprocessing stage for supervised learning models. The high-level PDC features generated are based on Ombui et al, (2019) hate speech annotation framework that is informed by the triangular theory of hate (Stanberg,2003).  Results obtained from frequency-based models using the PDC feature on the annotated dataset of ~48k short messages comprising of tweets generated during the 2012 and 2017 Kenyan presidential elections indicate an improvement on classification accuracy in identifying hate speech as compared to the baseline

best feature performance in codeswitched hate speech texts
abstract
How well can hate speech concept be abstracted in order to inform automatic classification in codeswitched texts by machine leaming classifiers? W We explore different representations and empirically evaluate their predictiveness using both conventional and deep leamnin algorithms in identifying hate speech in a -48k human-ane -nnottaad dataset that contain mixed languages, phenomenon common among multilingual speakers. This paper espouses novel approach handle this challenge by introducing a hierarchical approach that employs Latent Dirichlet Allocation to generate topic models that feed into another high-level feature set that we m PDC. PDC groups similar meaning words in word families during the preprocessing stage for supervised learning models. The high-level PDC features generated are based on Ombui et al, (2019) hate speech annotation framework that is informed by the triangular theory of hate (Stanberg,2000 2003). Results obtained from frequency- rrsuency-based based models using the PDC feature on the annotated dataset of -48k short messages comprising of tweets generated during the 2012 and 2017 K Kenyan presidential elections indicate an improvement on classification accuracy in identifying hate speech as compared to the baseline
Keywords: Hate Speech, Code- seideswitchhng, feature selection, representation tion learning
I. introduction
Identifying hate speech in short text messages retrieved from social media is challenging classification task. This data is often unstructured and noisy, coupled with the different dimensions of big data which include massive volume, high-velocity, questionable veracity, complex variety, and impactful to society(Buraa & Williams, 2015). The negative ripples of hate speech on social media, easily amplified across time, geographical and legal jurisdictions, not only abrogate user experience but have also been shown to turn into real-life violence in some cases (Hatzipanagos, 2018). This phenomenon is closely associated with online harassment, cyberbullying, body shaming and other hateful attacks targeting individuals or groups the basis of belonging to a protected characteristic like race, ethnicity, religion, gender, etc.
Social media companies phenomenon. Evidently, all social media networks have user policy on hate speech content on their platforms. However, most of these companies mostly rely on users to flag such content which then goes through some element of manual review to ascertain whether it violets the user policy on hate speech content(Sandler, 2018). Subsequently, decision is made to either to block the content, terminate the account, or take some other predefined action. This approach apparently is unfeasible in the light of big data.
increasingly under pressure by various stakeholders to better respond to the
The research community has responded well over time, clearly by the growing number of research activities including workshops, conferences, software applications, and publications in the area of automatic hate speech identification (W (amer and Hirschberg.2011 Kwok and W ang, 2013; Gitari et al., 2015), offensive language detection (X and Zhu, 2010;) and other related areas like cyberbullying (Cynthia et 2015), fake news detection, sentiment analysis, and stance detection. However, most of the studies have focused on European languages, especially English, whereas there are so many other languages used on social media platforms, especially regional and resource-scarce languages like Swahili which is lingua franca in East and part of central A frica. Besides, monolinguals form about 40% of the world population, whereas bilinguals, multilinguals, and polyglots form about 60%. (A nsaldo et al., 2018) We postulate that this statistics is increasingly being mimored on social media with the evidence of codeswitched language in datasets, for example the dataset that was built in our study that comprises of -400k tweets collected during the 2017 presidential campaigns in Kenya. A pparentlyy majority of K Kenyan anyas are multilingual, speaking their mother tongue (L1), Swahili as the national language(L2), and English as the official language(L2) (Muaka, 2011).
Therefore, this study bridges the gap for codeswitched language datasets in regards to automatic hate speech identification. To the best of our knowledge, this is the first study to collect and build a classifier for a codeswitched language dataset, specifically in English, Swahili, Sheng (slang) and some instances of words from native languages like Gikuyu and Luo. An example text message is,
"Hawa waarabu wanatorture kenyans wadhani sisi ni madogy zao.... Being our employers doesn't mean u own us ." [Translation: "These Arabs are torturing Kenyans thinking we are their dogs... Being our employer does not mean you own us." "]

---

In general, this study investigates the performance of various features various machine learning algorithms for hate speech identification in codeswitched language dataset. Both, conventional and deep learning features and learning algorithms were explored on -48k human- annotated dataset using supervised machine learning approach. We actualize the PDC high-level features by mapping them into tf-idf vector representations for the task of hate speech identification. Several experiments using the PDC features prove their effectiveness in capturing hate short text messages, with Passion (P) and Distance (D) components being the most salient with accuracies of 74.3% and 74.2% respectively. The PDC combined feature set is compared to the conventional features and the human annotation for the same dataset as baseline.
II. related work
There are several studies that have been conducted in relation to automatic hate speech detection. These can be classified into two main groups: those dealing with general hate speech as binary classification task and those as multi- class task. There are more studies in literature dealing with the phenomenon as a binary classification problem whereby the focus is on identifying a specific type of hate speech like racism (Lozano et.al, 2017, Tulkens et al., 2016;K wok & W Wagg 2013), anti-Semitism (W Warner& & Hirschberg, 2012) or neither. For the second class, the argument is that there is need differentiate the different types of hate speech hate speech from offensive messages because conflating them does not match the ground truth. (Davidson et al., 2017: 2017:Badjatiya et al., 2017; Fortuna, 2017).
There are various features that have been used previously in the detection of hate speech in social media text messages including demographic information of the users like the gender, location, (Waseem and Hovy,2016), syntactic features like the length of the message, part-of-speess tags as features and other high-level features, with dictionary lists and part-of-speech tags being popularly used. toe low-level features, the popular feature among the conventional learning algorithms has been the use of word frequency features like Bag-of- of-W ords (BoWs), s), n-grams (both at character and word-level) and T Term Frequency- Inverse features. W ord embeddings as features for deep learning neural network algorithms have increasingly become popular in hate speech detection studies( B Bdjjtiiya et al., 2017; Hasamuzzaman et al., 2017; Djuric al., 2015). The popularly used pre-trained embeddings include Global V Vectors (GloV e), FastT ext n- grams, and ord2V ec text representations, at both character, word and sentence levels. The general feature usage frequency in automatic hate speech studies is well summarized in figure 1.
Innerrs Document Frequency (tf-idf)
Feature Frequency
12
10
8
4
2
BOW
Brown clusters
Contextual
Dictionaries
E-binary count..
Hashtags
Length
Mentions
n-Grams
Num of...
Othering
Paragraph2Vec
Pattern matching
POS
Punctuations
Retweets
Rule- based.
Sentiment.
Template-..
Text features
TF-IDF
Tone analysis
Topic modelling
Topic similarity
Typed...
URLs
User features
Word2Vec
Wordsense...
figure 1: Feature usage frequency in literature
Some key observations from studies in literature include the popularity of conventional machine learning algorithm in learning classifiers and that English has been the most popular language of the datasets. A ccording to available literature there very few in other languages like Dutch (Tulkens et al., 2016), A Amhaiic (Mossie W Wang, 2018), A rabic (Al-Hassan & Al-Dosari, 2019) and none in Swahili. In addition, text messages (tweets) from Twitter have been popular with most of these studies. Besides, racism has been the most published type of hate speech. However, hardly any of the studies have publicly shared their full datasets apart from Davidson (2017).
III. method
The general methodology that was used to build the classifier models for hate speech classification in short text messages in this study was based on the CRoss- CRoss-Industry Standard Processes for Data Mining (CRISP- DM). Therefore, this study's pipeline entailed problem understanding, data acquisition and preparation, feature engineering, modelling, and evaluation.

---

In order to get deep understanding of the hate speech phenomenon, systematic literature review was conducted whereby several related studies and hate theories were examined. In addition, a content analysis was done on various definitions of hate speech as found on user policy guidelines of leading social media networks, and legal definitions of hate speech on constitution documents of various countries, with a focus on the case of K Kenya. This was further informed by the need of automating the monitoring for hate speech on social media by key govermment agencies in K Knyy like the National Cohesion and Integration Commision, which is in charge of matters related to hate speech(NCIC) primary Internet Service Provider all tertiary learning institutions in the Country (K enet, 2018), and other national security agencies. From this phase, a working definition of hate speech was derived, based on the section 13 of the NCIC Act of 2008 i.c. Hate speech is any message that incites hatred towards a target, whether person or a group of people, on the basis of belonging to protected group characteristic like ethnicity, race, religion, etc.
2019), the K Kenya Education Network that is the
A. data
There were 397, ,,555 raw text messages collected. These mainly consisted of tweets from the 2017 general election in Kenya, including the repeat presidential elections held later the same year. A dditional tweets from the 2012 general elections were also scrapped to form a formidable raw corpus size. This specific period was informed by existing research which indicates that there is often a higher volume of tweets generated during trigger events and not much thereafter (King & Sutton, 2013). Unlike text messages from other social media networks, tweets were purposively chosen because they are often topically structured, publicly available, and programmatically accessible via Twitter A Pls, python tweet collection libraries, and even using custom- -built crawlers. Moreover, Twwitter features that enable users to anonymously engage in public discourse over topical issues in real time makes it a conducive platform for hate speech propagation. Besides, several previous studies automatic hate speech detection have used tweets (K wok & W ang. (B & W Williams, 2015)(W aseem & Hovy, 2016). The main data acquisition strategy employed was bootstrapping, starting with a seed of known hateful keywords(W aseem, 2016) phrase pattems (W (Wiliiam W Wamer& & Hirschbery, 2012), then followed by apparent hateful hashtags during the presidential elections period. In addition, tweets from pro- hate speech Twitter user accounts (K & W Wang, 2013), especially bloggers and politicians who often post content bordering hate speech were collected.
Subsequently, this raw dataset was processed and culminated with clean dataset of kkkk tweets. The data cleaning process included lowercasing, the removal of tweets with unprintable characters, that is the first thirty- two character in the 7-bit ASIII code, removal of non- alphanummericcc and duplicate tweets, retweets, emojis, and advertisements that often ride on trending hashtags. Examples of tweets with unprintable characters include "À T """v"""**" and with advertisements include #NoRefomsNoElection A pple launch iPhone X their most expensive phoneaē?".. Moreover, actual mentions were replaced with "USERNAME" tag, and hashtag symbols replaced with "HASHTAG".
In addition, tweet character length was used. For example below tweets with a length of less than characters were removed. These comprised of tweets with only one or tw-characcters, with less than 3 word replies, which apparently did not meet the annotation scheme's 's threshold for the predefined classes of this research. Examples of these include "c", "ok", "DAAMN!", "I will". For other very short messages, they could not be contextualizen because the referenced message was not available or they comprised of only numbers, one-word acronym, m, or a symbol. Examples of these include "2S""" "mhh..ntt", "#".. Although some words, whether unigram or bigrams or trigrams, were offensive, they did not have an identifiable target of hate and therefore got dropped too using the criteria of "less than 4' word length. For example, "F#tkk you all!". In regard to tweet length, the conventional tweet message length has been 140 characters, which now has been doubled to 280 characters. In the contrary, from the observations made in this study, the longest tweet had 991 characters with two long URLs as part of the tweet message. This exception is explained by twitter's 's design that puts limit of 280 characters only the message, whereas anything else thereafter e.g. the URL address can be of any length. Given that the length of a message could probably be an informative classification feature (Davidson et al., 2017 7) for the study, only the message length was considered, and therefore all the URL part of the tweets were dropped.
Message Length
15000
10000
5000
25
49
73
97
121
145
169
193
217
241
265
289
313
337
361
Series1
Series2

---

figure 2:
Tweet length
B. data annotation
Using convenience sampling method, forty human annotators comprising of 80% undergraduate CS students and 20% staff from the school of science and technology at A frica Nazarene University (ANU) were recruited to help in the annotation exertise. The gender was relatively balanced, the average age was 23 but the nationality skewed towards K Kenyans. The skewness was informed by the need to have annotators who could easily interpret the codeswitched nature of the collected corpus (mostly in English, Swahili, and some other native languages). An initial training based on the annotation scheme was conducted and the team given sample messages to annotate using a web-based annotation portal developed by the research team (ref). Based on ththe annotation performance and commitment for that specific session, the team was trimmed to 27 annotators. Feedback from the first session was used to to enhance the annotation portal. A second training session was conducted and the team contracted to annotate at least 3k messages within week. The annotation portal was redesigned to have each tweet annotated by any three random annotators unlike during the first session whereby each tweet had to be annotated by at least. one subject matter expert (SME) and two novice annotators. The new design was infomed by the slow annotation process in the first session and the need to expedite the annotation process in order to have a sizeable labelled dataset for training our classifier.
C. Features
There were four primary features that were used in this study. These are the Bag-of of--of-Wonnn ords (BoWs), Term Frequency-Invern Document Frequency (TF-IDF), word embeddings, and the Passion- D Distance- Silrance-Comminne Commitment (PDC) features. Other features that were experimented with include Part-of- of--peechhppehhhe tags (PoS), and topic models as features. The BoWs features are basically frequency counts of term occurrences in tweet, were derived using the count vectorizer in Scikit-lear machine learning library. The TF-IDF vectors features are used to compute the relative importance of a specific term in the tweet and the entire dataset. There were 3 groups of TF-IDF F vectors that were derived based on the input token level, that is word, n-grams and character levels. Basically a matrix was generated for the TF-IDF score of each word in the different tweets, another one for the different word combinations (N-grams), and lastly one based on each character in the dataset.
Word Embeddings as features were based on GloVe e pre- pre-trained embeddings on the 100d file of about 1 million word vectors. The tweet dataset was tokenized and each token mapped to their respective embeddings, basically performing transfer learning. Topic Models as high-level features were intentionally used for data exploration and as an automated process to inform the specific words to include in the proceeding phase to generate the PDC family features. Here, Latent Dirichlet. A llocation (LDA) was used to generate 23 key topics from the hate speech dataset. These are as shown figure 3.
PDC features are psycholinguistic features derived from the 3 dimensions of hate as explicated by the triangle of hate theory (Sternbery & Stembery, 2008). As high-level features, PDC espouses hate speech in 3 primary word-families that are meaning-baat based and language independents. Therefore, the language list can grow or shrink by adding or removing similar meaning words in different languages in the respective families. Passion word family consist of words that express emotions of anger, fear, disgust and contempt. These include threatening, abusive, derogatory, and other offensive words directed towards a target person or groupp that belong to protected characteristics like race, ethnicity, religion, etc. An example message is "to hell with all <group>. They need to be swept from this country". Negative polarity and Sentiment analysis has been used in previous studies to detect passion instances (Chen et al., RRazaiia et al., 2010). Distance word family consists of words that express social distance or proximity in inter-group or persons relationships, what in also referred to as "ottherin" language (Bumap & Williams, 2016). This is often indicated by a high frequency usage of pronouns (Semin, 2009) (Coupland, 2010). For example, "us" ". "hhmm" they". "we" "ouu" ", etc. An example of an actual tweet is "Kambas also do not make good the are Cowards". Commitment word family consists of words or phrases that commit to blatantly hate on another person or group by devaluing. This can either be by referring to them using object, insect or animal names, or basically perceiving others as less superior, less mature, or less human (Haslam, 2006). Moreover, this includes some of the code names only known and used by the in- group to refer to the members of the out-group, An example tweet is "Kikuyus Are Enemies of Luos Stop Making Music With This Cockroaches".
D. Machine learning classifiers
Both conventional and deep learning classifiers were used to learn the classifier models. The conventional machine learning algorithms included the Naive Bayes (NB), Linear Logistic Regression (LLR), Support V Vector Machine (SV K-Nearest Neighbour (KNN), and Decision Tree (DT). In addition, Bagging and Boosting models, that is Random Forest (RF) and Extreme Gradient B oosting (X gb) were used respectively. In regard to deep leaming, Convolutional Neural Networks (CNN) and Hierarchical A ttention Network (HAN) were explored. There was minimum parameter fine tuning for the deep learning models. All the models used in the experiments were based on Scikit-leam machine learning models. A good summary of these and their respective classification performance on various features is well summarized on table 1.

---

E. Evaluation
The dataset was split into training and testing datasets: 80% for learning the classifier, and 20% for evaluating the performance of the trained classifier model. There were two key evaluations: evaluation in terms of the inter- annotator reliability score, and the evaluation of how discriminative each feature would be in the text classification task. The Krippendorff's alpha score was used to measure the inter-annotaly reliability. Both conventional classifier training algorithms and neural network algorithms were employed train the text classifier models. The best model was selected using grid search with 5-fold cross validation. ccuracy and F- measure based on precision and recall metrics were used to evaluate the saliency of each feature. For the deep learning algorithms, the iterations were set with an epoch value of 15.
IV. W.Resulss
Topic modeling for the short text messages using the Latent Dirichlet Allocation (LDA) approach derived 23 topics based on the Hate speech class. These are as shown in the 23 rows in figurel. The green cells indicate a legally protected characteristic, in this case the ethnic group names in Kenya and the nationality. The purple cells are individual names, mainly the presidential contenders/polittcians and one popular blogger. The blue cells also groupings but not falling under the protected characteristic category. These include, police, govemmenn, country and nation. The yellow cells indicate the "distancing" or "othering" features. The red cells indicate the "passion" features.
Took:
Kikuyc
Vilry
tribal
trico
country
Wint
Toplc
Luacs
trBal
They
U2huna
why
Lob
Toplc
hate
12eech
passion
Iove
Padka
resson
Kamba
dont
540
Lkhya
Tock
who
Batyes
Kluys
reagph:
Topk
Ryakand
Rada
W6B
19se
Topk
bane
prys
Topk:
11
food
They
Tosk
12
Kibayo
Mihroo
KBB
hoard
Lukin
Topk
18
Jest
ciid
elecice
04pport
Ruto
hate
Yoor
world
$0M
Toplc
14
Tayl
muth
Tyskand
Feal
comking
hdlp
10
Topk
15 people
Idam
Killors
Santhmm
Iniice
Topk
Mossal
ceantry
go-ng
Mara
hear
Free
communiiy
Topk
17
They
When
won
Topk
Keaye
wky
Topic
23
Took
25 people
shull
ipood
obmn
figure 3. Topic modeling for hate speech class
Four primary features were experimented on seven conventional and two deep leamiigg machine learning models. The model performance terms of accuracy was recorded as shown with the values in table 1. The table rows represent the features whereas the columns represent the respective machine learning algorithms.
Table 1: Feature performance across various ml algorithms
FEATURE
Machine learning A Igorithm
Name
Naive
Linear
SVM
KNN
DT
Bagging
Boosting Xgb
HAN
CNN
Bayes
L Regression
RF
1.BoW
0.720
0.743
0.741
0.706
0.739
0.737
0.743
2.FF-IDF
Word level
0.747
0.752
0.741
0.732
0.738
0.735
0.741
N-gram level
0.742
0.742
0.746
0.712
0.736
0.710
0.736
Character 0.739
0.751
0.725
0.738
0.724
0.743
level
3.Word Embeddings 4. 4.PDC
0.600
0.672
0.741
0.741
0.740
0.735
0.736
0.736
0.736
Features
The word embeddings as features were based on the GloVe vector representations (T (Twitter.27BB 27..100B dataset) with 1193514 word vectors. These were only used for the HAN and CNN models and not on the other models as shown in table 1.
V. discussion
Topic models as high-level features can be used to inform our PDC high-level features, which in turn can be used to generate informative lower-level features for hate speech classification. For example, from figure 2 in topic (row no. 1), "Kikuyus" which is the largest single ethnic group Kenya is closely linked with "thieves", " K Kenyans", "tribal" and "Uhuru". Generally, most of the hate related messages targeting Kikuyus had to do with labelling them as thieves, Kenyans, tribal, and had some relation to one of the presidential candidate "Uhuru".

---

In topic 2(row no.2), the "Luos", another large ethnic group in Kenya was closely linked with "kill", "tribal" "Uhuru". " "why". ". and "police". Hate messages along this topic had to do with questions ("why") of police killings of members of the Luo group during the campaign period. Another interesting topic was no. 9 which included "Wajahuo", which the Swahili translation for Luo ethnic group, is closely linked with "mawe" (stones), and (foooing A lot of the hate related messages targeting the "Wajaluo" were castigating them, "" "wajinga" for the violence and property destruction caused as a result of ttone-throwing. Generally, the use of psycholinguistic features in the PDC format can further be populated by other terms specified on LIW C (Tausczik & Pennebaker, 2010) to help characterize hate speech text at the high-level.
In regards to general feature performance, the TF-IDF feature outperformed all the other features, particularly at word-level The deep learning algorithms performance was lower than the conventional models. The lowest performing algorithm was the HA N. Results from the deep learning algorithms could get better with better hyper parameter tuning and more data.
The best algorithm was the linear logistic regression based on TF-IDF word- features that yielded an accuracy of 75.2%.
Existing tools and techniques in representation learning have mainly focused on one language, with English being the most popular. For example, exiting pre- pre-trained embeddings are mostly in English and other European languages. This becomes challenge for example when building classifiers to handle hate speech in A frican and other non- -un-eurpeann codeswitched datasets. From the extensive experiments in this study using both conventional, shallow and deep learning algorithms, is apparent that the conventional algorithms still perform relatively better on smaller datasets as compared to deep learning algorithms. In addition, the existence of codeswitched text. negatively impacts the performance of most conventional classifiers. The conventional classifiers are trained to handle text messages, often limited to one language. Therefore, the expected action by the classifier drop terms that are incomprehensible, just would for most unseen words during training. In this regard, there is need for a better approach to handle codeswitched language datasetss
VI. conclusion
Linear logistic regression algorithm trained on word-level TF-IDF achieves the best accuracy performance over our codeswitched language dataset. Besides, good results can be achieved inexpensively by using hierarchical structures for high-level features to better inform low-level features selected for training machine learning algorithms. In this study, we have explored the use of topic models as features to inform the PDC feature set which achieved competitive accuracies codeswitched dataset of -48k tweets and is espoused to perform equally well for multilingual datasets, especially where resource- scarrceesccceeeeeeeeeee languages are involved. Future work will involve experimenting with bigger datasets to see how well the approach scales and compare this with conventional features that have been prone to the curse of dimensionality and overfitting.
shown in table 1. This approach has worked well for a small
A cknowledgment
This study was partly funded by the Kenya Education Network (KENET) under the 2018 CSIS dini-graats research projects.
References
Al-Hassan, A., & Al-Dosari, H. (2019). Detection of Hate Speech in Social Networks: A Survey on Multilingual Corpus. 6th International Conference on Computer Science and Information Technology.
A Assalo,, A. I., Marcotte, K., Scherer, L., & Raboyeau, G. (2018). Language therapy and bilingual
aphasia: Clinical Implications of psycholinguistic and neuroimaging research. J Jurnal of Neurolinguistics, 21, 539-55.1 Retrieved from http://ilanguages.oug http://aangnggge.oob/liiggguggaa wrg/bilinggull..pp. php
Badjatiya, P., Gupta, S., Gupta, M., & V Vama, V. (2017). Deep Learning for Hate Speech Detection
in Tweets. 2017 International World Wide Web Conference Committee.
Burnap, P., & Williams, L. M. (2016). Us and them: identifying cyber hate on twitter across multiple
protected characteristics. EPJ Data Science, 1(5).
Buraa, P., & Williams, M. L. (2015). Cyber Hate Speech on Twitter. An pplication of Machine
Classification and Statistical Modeling for Policy and Decision Making. Policy & Internet, 2(7), 223-242.
Chen, Y., Zhou, Y., Zhu, S., & Xu, H. (2012). Detecting offensive language in social media to protect adolescent online safety. The Fourth ASE/IEEE International Conference on Social Computing (SocialCom 2012). Amstadam.
Coupland, N. (2010). "Other" representation, Society and Language. John Benjamins Publishing.

---

Davidson, T., W araalleyy D., Macy, M., & Weber, (2017).
A AutomatedHateSppecii Avontteccaaasspeccooooooottteeee
ICWSM.
Djuric, N., Zhou, J., Morris, Robin Grbovic, M., Radosavljevic, V., & Bhamidipati, N. (2015). Hate speech detection with comment embeddings. In Proceedings of the 24th
InternationalC onferenceon WorddWWdeeebdbbbb (tweebwwWWW55555 29 29-30.
Fortuna, P. (2017). Automatic detection of hate speech text: an overview of the topic and dataset annotation with hierarchical classes. University of Porto.
Gitari, D. N., Zuping, Z., Damien, H., & Long, J. (2015). A lexicon-basse approach for hate speech J ournal of Multimedia and Ubiquitous Engineering, 4(10), 215-230. Journal
Hasamuzzaman, M., Dias, G., & W Way, A. (2017).
DemographicW
Proceedings of the The 8th International J oint Conference on Natural Language Processing, 226-936.
Haslam, N. (2006). Dehumanization: An integrative review. Personality and Social Psychology
Review, 10, 252-64.
Hatzipanagos, R. (2018, November 30). How online hate turns into real-life violence. The Washington Post. Retrieved from https://www. https://www.washingg.
rea--life--vvoleeeee www.oleenncoooorddrrtttoooo Knnet. (2018). Kenya Education Network. Retrieved September 16, 2016, from https://cert.keee https//cert.kenet.or.kksss www.eeeeet....oooeee/e/eee.eeeee
King, R. & Sutton, G. M. (2013). High Times for Hate Crim:: Explaining the Temporal Clustering of Hate Motivated Offending. Criminology, 51),,, 71 14.
Kwok, I., & Wang, Y. (2013). Locate the hate: Detecting tweets against blacks. AAAL Mossie, Z., & W ang, J. (2018). SOCIAL NETWORK HATE SPEECH DETECTION FOR AMHAIIC LANGUA GE. COMIT, 41-55.
Muaka, L. (2011). Language Perceptions and Identity among Kenyan Speakers, Proceedings of the
40th Annual Conference on African Linguistics.
NCIC. (2019). Functions of the Commission. Retrieved September 16, 2019, from of tom-commmiiiiion
https://cohesion.or. wwt.wo.....eeeeeeeeeem wtw.//inw....p//bbottuuusseee
Razavi, A., Inkpen, D., Uritsky, S., & Matwin, S. (2010). Offensive Language Detection Using Multi-level Class fication. Springer, 1627.
Sandler, R. (2018, July). Facebook has apologized for flagging parts of the Declaration of Independence as hate speech. Business Insider. Retrieved from http:://www..ussinssssdddeee cttp///fcebooooddcaaaatttooooo of offiindeenndece- hate- mate-sseechh2211
7?r=US&RR=T
Semin, G. R. (2009). Linguistic Markers of Social Distance and Proximity.
Sterbbery, R., & Sternbery, K. (2008). The Duplex Theory of Hate I: The Triangular Theory of the Structure of Hate. In The Nature of Hate. Cambridge University Press, 51-77.
https://doi.org// /ttp://.00///BB0708111899
Tausczik, R., & Pennebaker, J. W. (2010). The Psychological Meaning of Words: LIWC and Computerized Text A nalysis Methods. ournal of Language and Social Psychology, 1(29).
Tulkens, S., Hilte, L., Lodewyckx, E., V Verhoeven, B., & Daelemans, W. (2016). The A utomated Detection of Racist Discourse in Dutch Social Media. CORR, Abs/1 608.08388. W Warner, W., & Hirschberg. (2012). Detecting Hate Speech on the Work World Wide Web. Language in Social Media (LSM 2012).
W Waseem, Z. (2016). Are Y ou a Racist or Am Seeing Things? A nnotator Influence on Hate Speech Detection on Twitter. EMNLP Workshop on NLP and CSS, W a'aseem, Z., & Hovy, D. (2016). Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of NAACL-HLT, 88-93.


Reviews:
[{"review_id": "r1gV_dm2KB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper compared several classification methods, including deep neural networks (DNN), to identify hate speech texts. Mainly the data was corrected from twitter. The data was prepossessed to deal with by popular classification methods. The LDA and PDA are used to construct the identifier with high accuracy. Finally, Practical data was used to assess the performance of several machine learning algorithms.\n\nThe research topic is important from the sociological viewpoint. I'm not sure whether this paper suits to the publication from ICRL. Besides that, the authors did not show any technical insight into the numerical results. That is, can the authors explain the reason why the linear logistic regression with TF-IDF features outperformed all the other methods? Overall, this paper did not provide any useful knowledge, while this paper introduced some statistical methods and showed numerical results. I recommend the authors to add more beneficial insight and to submit the paper to other conferences that deals with sociological issues.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "SyxCb3k6FH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I'm sorry to say that this paper is not ready for publication.\n\nI think it's an important area and the dataset collected could be quite valuable for tackling hate speech.\n\nThe paper does not follow the style guide, is full of typos or 'kkkkk' tokens indicating missing values. The first sentence of the abstract is not grammatical. Codeswitched needs to be mentioned more specifically in the introduction.\n\nI couldn't easily find statistics about the dataset, especially in terms of language breakdown. How many of the tweets were multi-lingual? For pure-english tweets, I would be interested in this dataset being split out as a sub-dataset, as I would heavily bet the sota method for classifying hate-speech would be to fine-tune a BERT model on these labels. We need a table of dataset statistics.\n\nWe need a mathematical definition of PDC that is made very explicit in the paper, there is too much prose, I did not have time to do the background reading of the linked papers to understand this sociological theory of hate speech.\n\nThe paper did not feel sufficiently anonymized. I would anonymize the university used to create the dataset and the funding agencies that supported the research in subsequent submissions.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgWdenDOr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Comments: \n\n-This paper considers codeswitched hate speech texts from an NLP perspective.  The dataset considers mixed languages.  \n\n-Focuses on kenyan presidential election.  \n\n-Paper has severe formatting issues as well as simple issues like capitalization.  Additionally many plots are rather unattractive (seem to be produced using Excel or Google Sheets, whereas generally something like matplotlib or seaborn is preferred).  \n\n  -The paper puts a lot of examples into the main text, whereas these are usually put into the appendix, or only a few examples are placed in the main text.  Usually the main text focuses more on higher level analysis.  \n\n  -Figure 2 is formatted incorrectly (the caption runs on to the next page)\n\n  -I appreciate the effort that went into data annotation as well as the disclosure of the demographics of annotators.  \n\n  -Table 1 should make the metric much clearer (it's mentioned in the main text, but it should be in the caption too, also the best performance usually should be bolded)!  Generally TF-idf features or PDC features seem to have the best performance.  The performance of the CNN does not seem very strong.  I think a simple RNN based approach might also be worth considering.  It would also be worth analyzing if the differences between the methods is attributable to underfitting or overfitting.  \n\n  -If the paper is proposing a new task with many baselines, it's also important to release the dataset and code in my opinion (I believe ICLR allows this to be done in a de-anonymized way).  \n\nReview: \n\nThis paper deals with an important problem in social media analysis.  With the spread of hate speech and hate crimes by rioting separatists in Hong Kong as well as equally hateful attacks on Chinese people in the west, I think that this is an issue that deserves more attention in our community.  Unfortunately this paper needs more polish to be appropriate for ICLR.  It also might be better suited to an NLP focused conference (such as ACL, EMNLP, or NAACL) although I think if the technical contribution is clear enough it could be suitable for ICLR as well.  \n\nI think the big things to focus on would be including more baselines, improving polish in the paper, and providing a clearer high-level analysis of the dataset (with specific examples mostly left for the appendix).    ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on hate speech detection and compares several classification methods including Naive Bayes, SVM, KNN, CNN, and many others. The most valuable contribution of this work is a dataset of ~400,000 tweets from 2017 Kenyan general election, although it is unclear whether the authors plan to release the dataset in the future.

The paper is difficult to follow, uses an incorrect ICLR format, and is full of typos.

All three reviewers agree that while this paper deals with an important topic in social media analysis, it is not ready for publication in its current state. The authors did not provide a rebuttal to reviewers' concerns.

I recommend rejecting this paper for ICLR.
Decision: Reject