Question: How to define OFFENSIVE (OFF) in the data?
Evidence: We developed the annotation guidelines jointly with an experienced annotator, who is a native Arabic speaker with a good knowledge of various Arabic dialects. We made sure that our guidelines were compatible with those of OffensEval2019. The annotator carried out all annotation. Tweets were given one or more of the following four labels: offensive, vulgar, hate speech, or clean. Since the offensive label covers both vulgar and hate speech and vulgarity and hate speech are not mutually exclusive, a tweet can be just offensive or offensive and vulgar and/or hate speech. The annotation adhered to the following guidelines:
Offensive tweets contain explicit or implicit insults or attacks against other people, or inappropriate language, such as:
Direct threats or incitement, ex: احرقوا> مقرات المعارضة> (“AHrqwA mqrAt AlmEArDp” – “burn the headquarters of the opposition”) and هذا المنافق يجب قتله> (“h*A AlmnAfq yjb qtlh” – “this hypocrite needs to be killed”).
Insults and expressions of contempt, which include: Animal analogy, ex: يا كلب> (“yA klb” – “O dog”) and كل تبن> (“kl tbn” – “eat hay”).; Insult to family members, ex: يا روح أمك> (“yA rwH Amk” – “O mother's soul”); Sexually-related insults, ex: يا ديوث> (“yA dywv” – “O person without envy”); Damnation, ex: الله يلعنك> (“Allh ylEnk” – “may Allah/God curse you”); and Attacks on morals and ethics, ex: يا كاذب> (“yA kA*b” – “O liar”)
Vulgar tweets are a subset of offensive tweets and contain profanity, such as mentions of private parts or sexual-related acts or references.
Hate speech tweets, a subset of offensive tweets containing offensive language targeting group based on common characteristics such as: Race, ex: يا زنجي> (“yA znjy” – “O negro”); Ethnicity, ex. الفرس الأنجاس> (“Alfrs AlAnjAs” – “Impure Persians”); Group or party, ex: أبوك شيوعي> (“Abwk $ywEy” – “your father is communist”); and Religion, ex: دينك القذر> (“dynk Alq*r” – “your filthy religion”).
Clean tweets do not contain vulgar or offensive language. We noticed that some tweets have some offensive words, but the whole tweet should not be considered as offensive due to the intention of users. This suggests that normal string match without considering contexts will fail in some cases. Examples of such ambiguous cases include: Humor, ex: يا عدوة الفرحة ههه> (“yA Edwp AlfrHp hhh” – “O enemy of happiness hahaha”); Advice, ex: لا تقل لصاحبك يا خنزير> (“lA tql lSAHbk yA xnzyr” – “don't say to your friend: You are a pig”); Condition, ex: إذا عارضتهم يقولون يا عميل> (“A*A EArDthm yqwlwn yA Emyl” – “if you disagree with them they will say: You are an agent”); Condemnation, ex: لماذا نسب بقول: يا بقرة؟> (“lmA*A nsb bqwl: yA bqrp?” – “Why do we insult others by saying: O cow?”); Self offense, ex: تعبت من لساني القذر> (“tEbt mn lsAny Alq*r” – “I am tired of my dirty tongue”); Non-human target, ex: يا بنت المجنونة يا كورة> (“yA bnt Almjnwnp yA kwrp” – “O daughter of the crazy one O football”); and Quotation from a movies or a story, ex: تاني يا زكي! تاني يا فاشل> (“tAny yA zky! tAny yA fA$l” – “again O Zaky! again O loser”). For other ambiguous cases, the annotator searched Twitter to find how actual users used expressions.
Table TABREF11 shows the distribution of the annotated tweets. There are 1,915 offensive tweets, including 225 vulgar tweet and 506 hate speech tweets, and 8,085 clean tweets. To validate the quality of annotation, a random sample of 100 tweets from the data, containing 50 offensive and 50 clean tweets, was given to additional three annotators. We calculated the Inter-Annotator Agreement between the annotators using Fleiss’s Kappa coefficient BIBREF23. The Kappa score was 0.92 indicating high quality annotation and agreement.
OFFENSIVE (OFF): Offensive tweets contain explicit or implicit insults or attacks against other people, or inappropriate language, such as: Direct threats or incitement, ex: é PAªÖÏ@ H@Q  ®Ó @ñ ¯Qk@ (“AHrqwA mqrAt AlmEArDp” – “burn opposition headquar_x0002_ters”) and   ¯A JÖÏ@ @ Yë @ñÊ J  ¯@ (“h*A AlmnAfq yjb qtlh” – “kill this hypocrite”).
Answer: OFFENSIVE (OFF): Offensive tweets contain explicit or implicit insults or attacks against other people, or inappropriate language, such as: Direct threats or incitement.