A Dataset for Investigating the Impact of Context for Offensive Language Detection in Tweets
Authors: Musa Nuri İhtiyar
Ömer Özdemir
Mustafa Emre Erengül
Arzucan Özgür
Venue: EMNLP 2023 Findings (2023)
Keywords: Offensive Language Detection
Twitter Dataset
Language Resources and Evaluation

Abstract: Offensive language detection is crucial in natural language processing (NLP). We investigated the importance of context for detecting such language in reply tweets on Twitter, where the use of offensive language is widespread. We collected a Turkish tweet dataset where the target group was unvaccinated people during the Covid period. Tweets in the dataset were enriched with contextual information by adding the original tweet to which a particular tweet was posted as a reply. The dataset, which includes over 28,000 tweet-reply pairs, was manually labeled by human annotators and made publicly available. In addition, we compared the performance of different machine learning models with and without contextual information. Our results show that this type of contextual information was not very useful in improving the performance of the models in general, although it slightly increased the macro-averaged F1-score of certain models.

A Dataset for Investigating the Impact of Context for Offensive Language Detection in Tweets
Musa Nuri ihtiyar, Ömer Özdemir, Mustafa Emre Erengül, Arzucan Özgür musa.ihtiyar, omer.ozdemirl,, mustafa.erengul, arzucan. ozgur) @ @boun.edu.tr Department of Computer Engineering, Bogaziçi University
Abstract
has also been shown to be very effective in detect- ing offensive language (Dai al., 2020; Zampieri et al., 2020; Mozafari et al., 2020). A similar trend can be observed for other languages. For exam- ple, Mubarak et al. (2023) used AraBERT (Antoun et al., 2020), the Arabic version of BERT, for Ara- bic. Similarly, BERTurk (Schweter, 2020) has been successfully used detect offensive language in Turkish tweets (Beyhan et al., 2022; Toraman al., 2022; Ann et al., 2023).
Offensive language detection is crucial in natu- ral language processing (NLP). We investigated the importance of context for detecting such language in reply tweets on Twitter, where the of offensive language is widespread. We collected Turkish tweet dataset where the tar- get group was unvaceinated people during the Covid period. Tweets in the dataset were riched with contextual information by adding the original tweet to which particular tweet posted as a reply. The dataset, which in- cludes over 28,000 tweet-reply pairs, was man- ually labeled by human annotators and made publicly available. In addition, we compared the performance of different machine learning models with and without contextual informa- tion. Our results show that this type of con- textual information was not very useful proving the performance of the models in gen- eral, although it slightly increased the macro- averaged F1- -core certain models.
Annotated datasets are needed to train or fine- tune machine learning models for offensive lan- guage detection. A number of datasets have been prepared for different languages and domains and made publicly available (Basile et al., 2019; Zampieri et al., 2020; ElSherief et al., 2021). A lim- itation of these datasets is that generally each tweet is labeled individually without considering contex- tual information. There are few studies that con- sider contextual information. Mosca et (2021) investigate the relative contribution of user informa- tion features in machine learning models by using explainability techniques. Cécillon et al. (2021) propose a traph-based approach to represent dia- log data from chat logs of an online game and use this representation for abusive language detection. Yu et al. (2022) define context as the previous com- ment in a Reddit conversation thread and show that such contextual information useful for detecting hate speech.
1 Introduction
Humans can communicate through language, which enables them to engage in many useful activi- ties, yet language might also be used for destructive purposes. One of the most critical examples of this is offensive language, which can be defined "any utterance which is blasphemous, obscene, indecent, insulting, hurtful, disgusting, morally repugnant, which breaches commonly accepted standards of decent and proper speech" ILsweInsider, 2023).
We hypothesize that similar contextual informa- tion may be useful for offensive language detection in tweets. As a motivating example, consider a reply tweet that states, "I fully agree." The cate- gory of this reply tweet (i.e., whether it is offensive or not) depends the previous context, i.c., the tweet to which it was posted reply. To inves- tigate the impact of such contextual information on commonly used machine learning-basee offen- sive language detection models, we collected and manually annotated tweet-reply pairs Turkish, a
The use of offensive language can occur on a variety of platforms, but is particularly common on online platforms such as Twitter. In recent years, several approaches have been proposed to automat- ically detect offensive language in tweets. Finc- tuning language models pre-trained with extensive data is considered the current state-of-the-arr for detecting offensive language. BERT (Devlin et al., 2019) is one of the most prominent transformer- based tranted language models for English and

---

low-resource language with limited datasets. One of the first tweet datasets for detecting offensive language in Turkish was developed by Çöltekin (2020). Recently, Beyhan et al. (2022) and Tora- man et al. (2022) also released tweet datasets for Turkish. However, none of these datasets consider contextual information.
March 2020, when the virus reached Türkiye, September 2022, when the pandemic was no longer on the agenda for most people on the planet. We used search by keyword with 16 different queries such as "aşIsız" (unvaccinated) and "aşı olmak iste- meyen" (those who do not want to be vaccinated) to identify relevant tweets. The keywords are phrases meaning "aşIsız" (unvaccinated) with different sin- gular/plural forms or spellings due the Turkish character related issues. The list of all keywords used in this study can be found in the Appendix.
We chose our domain the Covid-19 pandemic, which affected our lives in a number of different ways. Pandemics trigger fear and anger most people, leading to increased use of offensive lan- guage. Sharif et al. (2021) studied the detection of hostile statements in the context of the Covid-19 pandemic, and Bor et al. (2023) showed that such offensive language occurred against unvaccinated people during this period. Therefore, we selected unvaccinated people as our target group.
There were different options to search for the replied and reply tweet pairs. The first one was getting pairs where at least one of the 16 search keywords occurred in the reply tweet. We call this Dataset 1. Another possibility is that these keywords occur in the replied tweet. This case contains two subcases. The first case to have at least one of these keywords in a replied tweet, which itself is a reply to another tweet. We refer to this case as Dataset 2. Finally, the last case is to have at least one of these keywords in a replied tweet that is not itself reply to another tweet. This case is called Dataset 3. All three of these datasets were merged to obtain the final dataset.
The main contributions of this paper are twofold: (i) We collect and manually annotate a Turkish tweet dataset specific to the Covid-19 period and containing contextual information in the form of the replied tweet. (ii) We investigate the impact of such contextual information the performance of commonly used machine learning-based models for offensive language detection. The dataset and source code are made publicly available for future studics.!
Although conversations on Twitter could be ar- bitrarily long, we only looked at the previous tweet (replied tweet) to avoid unnecessarily complicated data format. In other words, all of the samples in our dataset are pair. Yet, we could capture any replied-reply couple related unvaccinated people as long least one of the tweets contains one or more of the pre-determined keywords. During the search, we collected tweet ID and tweet text information for both the replicd and reply tweets.
The rest of the paper is organized as follows. While Section 2 examines the collection and an- notation of the dataset, Section 3 focuses the experiments conducted to compare the machine learning models with and without contextual infor- mation. Finally, Section 4 discusses the lessons learned.
2 Dataset
Once the collection process was completed, we proceeded with labeling. The objective of the an- notation to obtain binary label indicating whether or not the reply tweet contains offensive language against unvaccinated people. Making ex- planations about specific points essential for this part. First of all, we decided to keep the task clear so that we could understand the impact of the con- text better, so using binary label looked like the best option, and we only looked at offensive lan- guage against unvaccinated people; in other words, even if reply tweet was offensive, against immi- grants for instance, labeled that as "not offen- sive against unvaccinated people" instead of "of- fensive against unvaccinated people". This was not because such offensive language was acceptable
We collected a dataset containing replied and reply tweet pairs. A reply tweet is tweet written in response to another tweet, while replied (weet is tweet to which another tweet has replied. Suppose a tweet T1 posted and then another tweet T2 is posted in response T1. In this case, T'1 is called a replied tweet and T2 is called reply tweet.
Our goal was to create a target group-specific dataset to enable the development of models capa- ble of detecting offensive language towards spe- cific target group. We selected unvaccinated people in the Covid 19 pandemic the target group for offensive language. We examined the period from
'https: htw.bt/ttttttutecc/// wttp:llccooobbutaaaee
2

---

but due to the fact that we wanted to have a single target group to make the problem more focused such that the effect of the context could be seen more directly. Solely focusing on the offensiveness of the reply tweet was done since the context rel- evant only for the reply tweet. That is, a pair where the replied tweet was offensive against unvacci- nated people, but the reply tweet was not offensive is categorized "not offensive" since we are only interested in the reply tweet's behavior.
for these labels, so we used the F1 and Cohen Kappa scores. We obtained 55.22% and 46.26%, respectively, for these metries.
After obtaining the annotations by two annota- tors for each pair of tweets, the annotations were examined. If there is consistency, then this was chosen as the ultimate label. the ultimate label is "ambiguous", is removed; otherwise, it is added to the final dataset. If there is inconsistency in the form of one annotator choosing "not offensive" and the other choosing "offensive", these cases are ambiguous; consequently, these were removed well. For the inconsistencies where one annotator chose "ambiguous", the third annotator looked the tweet pair and determined the final decision. If a label other than "ambiguous" was chosen, then it is selected as the last label. If not, it was re- moved. After several hours of this procedure, pairs with binary labels were obtained. In total, we ob- tained 28808 pairs. While 13478 of them came from Dataset 1, Datasets 2 and 3 contributed with 1515 and 13815 pairs, respectively. The final binary dataset has 27219 examples that are not offensive against unvaccinated people, denoted with 0, and 1589 examples which are offensive against unvac- cinated people which are denoted with since was representing the ambiguous case. The dataset is inevitably imbalanced since 94.48% of the pairs are labeled 0. Inter-annotatorr agreement for the dataset's last version Fl score and Cohen Kappa score. This time they were calculated as 95.21% % and 88.97%, which is significantly better than the initial version of the dataset. The final version of the dataset contain- ing the replied and reply tweet ids as well as the manual annotations is made publicly available for future 2
Which cases to consider offensive language is another crucial point to explain. Situations like swearing and insulting were the most obvious ones. addition, provocative words like stating that there should be a punishment, such as not being able to go outside or get into closed areas, with- out stating any exception or an alternative option, for unvaccinated people are included in this label. Also, want to express that quotations sim- ply stating an idea without using harmful language. like saying that "not getting vaccinated is wrong behavior," are not perceived as offensive language. Even we determine criteria, as we mentioned, for when to consider tweet as offensive. this field is inevitably subjective for specific examples. This is why at least two people annotated each pair in our dataset.
The annotation process was carried out as fol- lows. A general guideline for annotation tablished and provided to the annotators (i.e., three of the authors of the paper) and a training was per- formed by using sample examples. Each tweet pair was annotated independently by two annotators and a third annotator was used to resolve incon- sistencies. For each tweet pair, there were three label options, namely "not offensive against un- vaccinated people", "ambiguous". and "offensive against unvaccinated people". Although it is stated that the goal was obtaining binary labels, three op- tions were given in order to provide more flexibility to the annotators; however, the pairs whose final label is "ambiguous" " were removed from the final dataset since this would make the primary goal of the study more difficult to interpret which was ex- amining the effect of taking the replied tweet into account. While doing the annotation, totally un- related cases in the dataset, such unvaccinated fruits and vegetables owing to chosen keywords, were mostly cleaned even though a limited number such cases might be still existing in the dataset. We wanted to measure annotaart agreement
es-
measured using the
3 Experiments and Results
After completing the annotation of the dataset, we used it to train and evaluate various machine Iearn- ing models to detect offensive language against unvaccinated people. We randomly selected 20% of the dataset the test set. For each algorithm we used, we examined two different scenarios. the first, we used only the reply tweet, while in the second, we studied the impact of using the replied tweet in addition to the reply tweet on our models.
Ptt //gi thub. www./oooontoattieeeeeee CovidOf wwg.reeenssiiaaaggggggliitttttte
3

---

Method
Prec
Rec
F1
Method
Prec
Rec
F1
KNN (1)
20.56
41.12
27.41
BERTurk (1)
65.73
82.68
70.28
KNN (2)
20.84
40.79
27.59
BERTurk (2)
70.11
79.03
73.57
LR (1)
50.00
39.80
44.32
Table Results for deep learning models. (1) corre- sponds to the first scenario where only the reply tweet is used and (2) corresponds to the second scenario where both the reply and the replied tweet are used.
LR (2)
44.72
41.78
43.20
MNB (1)
65.32
26.64
37.85
MNB (2)
45.65
34.54
39.32
SVM (1)
50.76
44.08
47.18
SVM (2)
51.46
34.87
41.57
RF (1)
38.51
39.14
38.82
RF (2)
43.25
35.85
39.21
vlin al., 2019) obtained incredible success in the last years. Therefore, we decided look the performance of the Turkish version of the BERT model called BERTurk (Schweter, 2020) with and without replied tweet information. For the single tweet setting, we followed the classical procedure for fine-tuning where we used binary cross-entropy with Adam optimizer (Kingma and Ba, 2015) with 5x100= -5 learning rate. We did the hyperparame- ter optimization by looking at the validation set Fl score. For the case of using two tweets (the reply and replied tweet), the only difference was creating longer input string by combining the two tweets in the form of "Önceki tweet: replied_ tweet Cevap: reply_ Stelyy_ (in English, "Previous tweet: replied_twer _teett Reply: (macro-averagee scores) obtained the test set are summarized for the cases in Table 2.
Table Results for traditional models. For each model, (I) corresponds to the first scenario where only the reply tweet is used and (2) corresponds to the second scenario where both the reply and the replied tweet used.
3.1 Traditional Machine Learning Models
Simple machine learning algorithms might perform quite good for certain tasks. Therefore, we started with simple algorithms such Logistic Regression (LR), K-Nearest Neighbors (KNN), and Multino- mial Naive Bayes (MNB). Then we also used Sup- port Vector Machines (SVM) and Random Forest (RF). Since our dataset was imbalanced, we used downsampling to increase the performance of our models. In other words, we randomly selected a subset for the not offensive class while using all samples for the offensive class since it already had a limited number of samples. We had 1285 positive samples in the training set, so decreased the not offensive class 4500 samples, since too much reduction would cause a data scarcity problem. We used a tf-idf based vector representation for the tweets. The performance of the commonly used traditional machine learning algorithms is given in Table with the macro-averagee Fl score, preci- sion, and recall.
The results
Interestingly, this time the model that uses both the reply and the replied tweet performed better in terms of Fl score, yet the effect of taking context into account still limited. Even though precision improves, recall drops. The English translation of an example to explain this phenomenon is provided below. In this example, the reply tweet is offen- sive, while the replied tweet is not offensive. In this case, including the replied tweet as contextual information to classify the reply tweet misleads the model.
There are two main observations we can make with these results. These simple models are not able to perform well on this task. Even if we had used a majority classifier, we would obtain 50.0% recall, 47.24% precision and 48.5 58% Fl score. The inclusion of information from the replied tweets does not have significant impact the perfor- mance of the models and behaves more like noise.
Replied Tweet: read the National Anthem."
"Vaccination opponents mis-
"Go away the army of brain- less people to your village, you can't live in the metropolis without vaccine."
Reply Tweet:
For more example tweets where the inclusion of context (i.e., the replied tweet) is necessary for the correct classification of the reply tweet and where context could mislead the classifier, see the Appendix.
3.2 Deep Learning Models
Deep Learning models are top-rated in natural language processing. Especially the transformer- based ones (Vaswani et al., 2017).ll. like BERT (De-
4

---

4 Conclusion
use, the annotation of datasets is an essential step. which ought to be carried out responsibly. Despite the fact that we tried to use multiple annotators for the labeling process, developing better strategies are possible since some examples regarding offen- sive language are very subjective. The annotated data is shared based on Twitter's terms of use.
We prepared offensive language dataset for Turk- ish, where the number of such datasets is very lim- ited. Unlike most other tweet datasets where each tweet considered individually, we included the replied tweet as contextual information and investi- gated how this information affects the performance of commonly used machine learning models. Con- trary to our expectation, our results showed that this resulted in only slight improvement in the Fi-score for some models and did not significantly improve the performance of the studied models for offensive language detection in general. In theory, the previous tweet appears to contain important information. However, in analyzing our dataset, we found that most reply tweets have only a weak relationship to the replied tweet in terms of mean- ing. Moreover, interaction with other tweets is dominated by the use of other features on Twit- ter, such "like" or "retweet." Consequently, the use of information about previous tweets did not provide much contribution for offensive language detection in this study. Nonetheless, attempting to develop models specifically designed to consider information about previous tweets could lead to bet- ter performance and represents promising future research direction.
A Acknowledgements
This work is partially supported by the EU funded project entitled "Utilizing Digital Technology for Social Cohesion, Positive Messaging and Peace by Boosting Collaboration, Exchange and Solidar- ity" and by the Boğaziçi University Research Fund under the Grant Number 16903.
References
Wissam Antoun, Fady Baly, and Hazem Hajj- 2020. AraBERT: Transformer-bar based model for Arabic lan- guage understanding. In Proceedings the 4th Work- shop on Open-Source Arabic Corpora and Process- ing Tools, with a Shared Task on Offensive Language Detection, pages 9-15, Marseille, France. European Language Resource Association.
Inanç Anm, Zeynep Işık. Seçilay Kutal, Somaiyeh De- hghan, Arzucan Özgür, and Berrin Yanıkoglu. 2023. STU2023-NST Hate Speech Detection Contest. In 31 Signal Processing and Communications Appli- cations Conference (SIU), pages 1-4.
Limitations
While we tried various methods for detecting offen- sive language with and without replied tweet, we have not focused on developing a specific model which benefits from the previous (i.e., replied) tweet in the best way. Our goal was investi- gate the impact of contextual information on the performance of commonly used machine learning- based models. Therefore, even though we were not able to get significant improvements with contex- tual information, further research focusing on this subject is a promising direction to follow.
Valerio Basile, Cristina Bosco, Elisabetta Fersini, Deb- ora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. Semeval-200 2019 Task 5: Multilingual detection of hate speech against immigrants and women in Twitter. In Proceedings of the 13th International Workshop Semantic Evaluation, pages 54-63.
Fatih Beyhan, Buse Çank, Inanç Ann, Ayşecan Terzioglu, Berrin Yanıkoğlu. and Reyyan Yeniterzi. 2022. A Turkish hate speech dataset and detection system. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4177- 4185.
We examined the use of previous tweet for only single target group and language due to the labori- ous nature of the manual annotation process and the time limitations. The dataset can be expanded with other target groups and languages in the future.
Alexander Bor, Frederik Jørgensen, and Michael Bang Petersen. 2023. Discriminatory attitudes against unvaccinated people during the pandemic. Nature, 613(7945):704-711
Noć Cécillon, Vincent Labatut, Richard Dufour, and Georges Linares. 2021. Graph embeddings for abu- sive language detection. SN Computer Science, 2:1- 15.
Ethics Statement
Offensive language detection systems could be very useful for real-life uses. Because machine learning- based models are guided mainly by the data they
Çağri Çöltekin. 2020. corpus of Turkish offensive language on social media. In Proceedings of the
5

---

Twelfth Language Resources and Evaluation Confer- ence, pages 6174-6184, Marseille, France. European Language Resources Association.
Language Resources and Evaluation Conference, pages 2213-2225 2225, Marseille, France. European Lan- guage Resources Association.
Wenliang Dai, Tiezheng Yu, Zihan Liu, and Pascale Fung. 2020. Kungfupanda SemEval-2024 Task 12: BERT-BBased Multi- Tarning for Offensive Lan- guage Detection. In Proceedings the Fourteenth Workshop on Semantic Evaluation, pages 2060-2066.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Jacob Devlin, Ming-Wir Mii Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing, In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers). pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Xinchen Yu, Eduardo Blanco, and Lingzi Hong. 2022. Hate speech and counter speech detection: Conver- sational context does matter. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Hu- man Language Technologies, pages 591 8-5930, Seat- tle, United States. Association for Computational Linguisties.
Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çagn Çöltekin. 2020. SemEval-2020 Task 12: Multilingual Offen- sive Language Identification in Social Media (Or- fensEval 2020). In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1425 1447, Association for Computational Linguistics.
Mai ElSherief, Caleb 7iems, David Muchlinski, Vaish- navi Anupindi, Jordyn Seybolt, Munmun De Choud- hury, and Diyi Yang. 2021. Latent hatred: A bench- mark for understanding implicit hate speech. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods Natural Language Processing, pages 345-363.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
A
Tweet Pair Examples Regarding Context Information
A.1
An example where context is necessary for correct classification of the reply tweet
Law-Insider. 2023. Offensive language definition law insider. Accessed on June 18, 2023.
The English translation:
Edoardo Mosca, Maximilian Wich, and Georg Groh. 2021. Understanding and interpreting the impact of user context in hate speech detection. In Proceedings of the Ninth International Workshop on Natural Lan- guage Processing for Social Media, pages 91-102.
Replied Tweet: If we are closed at home again because of those who are not vaccinated, you will see curses that you have not seen so far in this account..
Reply Tweet: +1
Marzieh Mozafari, Reza Farahbakhsh, and Noel Crespi
2020. A BERT-based transfer learning approach for hate speech detection online social media. In Com- plex Networks and Their Applications VIII: Volume 1 Proceedings the Eighth International Conference on Complex Networks and Their Applications COM- PLEX NETWORKS 2019 8, pages 928-940 Springer.
A.2 An example where context does not matter
English translation:
Replied Tweet: may be against the necessity of vaccination, may be thought that the mask is not protective; however, there is no human side of walking as a group on a girl who works as a cashier under difficult conditions, entering a closed area without a mask, and causing fear and sadness.
Hamdy Mubarak, Sabit Hassan, and Shammur Absar Chowdhury. 2023, Emojis as anchors to detect Ara- bie offensive language and hate speech. Natural Language Engineering, page 1-22.
Stefan Schweter. 2020. BERTurk - BERT models for Turkish.
Reply Tweet: Those who are not vaccinated those who do not wear masks. I seriously don't understand what's wrong with this team. This team seriously litmus of intelligence.
Omar Sharif, Eftekhar Hossain, and Mo- hammed Moshiul Hoque 2021. Combating hostility: Covid-19 fake news and hostile post detection in social media. arXiv preprint arXiv: 2:Xi:22001.3220
A.3
An example where reply is not offensive but replied might mislead since it is offensive
Cagri Toraman, Furkan Şahinug, and Eyup Yilmaz. 2022. Large-scale hate speech detection with cross- domain transfer. In Proceedings of the Thirleenth
English translation:
6

---

Replied Tweet: Prof. Bingür Sönmez: Those who say they will get vaccinated are traitors, we will not allow them to get married with our girls A.4 An example where reply is offensive but replied might mislead since it is not offensive
Reply Tweet: At the point where the cardiovas- cular surgeon has come, we will not allow traitors who do not get vaccinated to get married with our girls.
English translation:
Replied Tweet: Vaccination opponents misread the National Anthem
Go away the army of brainless peo- ple to your village. you can't live in the metropolis without vaccine.
Reply Tweet:
B Keywords used for Getting Related Tweets
The following keywords were used in our search: aşIsız, asisiz, aşısızlar, asısizlar, aşı olmayan, ası olmayan, aşı olmayanlar, ası olmayanlar, aşı ol- mak istemeyen, olmak istemeyen, aşı olmak istemeyenler, ası olmak istemeyenler, aşı yapti-- mayan, ası yapturmayan, aşı yapturmayanlar, ası yapurmayanlar.
7


Reviews:
[{"review_id": "N2oYbnCivv", "reviewer": "Reviewer_xHbP", "summary": "", "questions": "- How were the human annotators trained to label the tweets, and how did you ensure consistency in their annotations?\n- Could you elaborate on why the contextual information did not significantly improve model performance in most cases?\n- Were there specific types of offensive language or subtleties that the context helped to identify?\n- Why was the specific target group of unvaccinated people during the Covid period chosen for this study?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments and constructive feedback. We briefly address the comments and questions below.\n\n**Q:** “Macro-averaged F1-score: The paper mentions a slight increase in the macro-averaged F1-score for certain models with context. It would be helpful to know which models benefited from the context and why.”\n\n**A:** The results in Tables 1 and 2 show that MNB, RF, and BERTurk benefited from the context information. However, the traditional ML models including MNB and RF obtain much lower performance compared to BERTurk with or without context information. We investigated the results obtained by BERTurk in more detail and identified three main types of cases where context information is helpful or where it misleads the model. These three cases are listed with one sample example each in the Appendix section.\n\n**Q:** How were the human annotators trained to label the tweets, and how did you ensure consistency in their annotations?\n\n**A:** A general guideline for annotation was determined and provided to the annotators and a training was performed by using sample examples. As explained in the Dataset section, each tweet was anno-tated by two annotators independently, a third annotator was used to resolve inconsistencies and tweet pairs with ambiguity were filtered by following the procedure described in the Dataset section of the paper. The inter-annotator agreement F1 score and Cohen Kappa score for the final version of the da-taset was calculated as 95.21% and 88.97%, respectively.  \n\n**Q:** Could you elaborate on why the contextual information did not significantly improve model performance in most cases?\n\n**A:** We were expecting the previous (original or replied) tweet to contain important information related to the class of the reply tweet. However, by analysing the results and the dataset, we found that the reply tweet in many cases is not a direct response to the original tweet, but rather provides the author’s independent opinion on the topic. Also, in some cases, while the reply tweet is in one category, the original tweet may be of the opposing category misleading the models. Example cases are provided in the Appendix. As the reviewer suggested, including other types of context information may be more helpful, which we plan to investigate as future work. Nevertheless, we believe the current dataset and the results will provide a useful basis for future work in this direction.\n\n**Q:** Were there specific types of offensive language or subtleties that the context helped to identify?\n\n**A:** We provided example cases in the Appendix, but we will conduct a more detailed analysis. We thank the reviewer for this point. \n\n**Q:** Why was the specific target group of unvaccinated people during the Covid period chosen for this study?\n\n**A:** We believe infectious diseases and vaccination/unvaccination will continue to be important topics and the created dataset may be useful for further analysis in other disciplines as well, such as social or health sciences.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The major contribution is three fold. 1) Dataset Introduction 2) Context-wise Enrichment 3) Comparative Analysis\n\nDataset Introduction: The data would be a valuable contribution to the community. The authors collected 20,000 tweet-reply pairs and manually label them. This dataset can serve a test bed for Turkish NLP tasks.\n\nContext-wise Enrichment: The authors enhanced the original tweets with corresponding replies. This thread based dataset can potentially help in understanding the nuances and subtleties of offensive language in a conversation.\n\nComparative Analysis: The comparison of machine learning model performances with and without contextual information provides a clear benchmark for future studies.\n", "reasons_to_reject": "Contextual Impact: The paper concludes that the contextual information was not very useful in improving the performance of the models. It would be beneficial to delve deeper into why this was the case. Is it because of the nature of the tweets, the models used, or some other factor?\nModel Details: The summary does not provide details about which machine learning models were used. A more detailed analysis of the models, their architectures, and hyperparameters would be beneficial for reproducibility and a deeper understanding of the results.\nTarget Group: While the choice of focusing on unvaccinated people during the Covid period is unique, it might also introduce biases. The paper should discuss potential biases and how they might affect the results.\nMacro-averaged F1-score: The paper mentions a slight increase in the macro-averaged F1-score for certain models with context. It would be helpful to know which models benefited from the context and why.\n", "questions_to_authors": "- How were the human annotators trained to label the tweets, and how did you ensure consistency in their annotations?\n- Could you elaborate on why the contextual information did not significantly improve model performance in most cases?\n- Were there specific types of offensive language or subtleties that the context helped to identify?\n- Why was the specific target group of unvaccinated people during the Covid period chosen for this study?", "paper_topic_and_main_contributions": "The paper discuss offensive language detection in reply of tweets with a focus on the context importance. \n\nThe main contributions of the paper are:\n\n1. The creation and public release of a new dataset: The authors collected a Turkish tweet dataset targeting unvaccinated people during the Covid period. This dataset consists of over 20,000 tweet-reply pairs, where the replies are enriched with contextual information by adding the original tweet to which a particular tweet was posted as a reply. The dataset was manually labeled by human annotators.\n\n2. An empirical study comparing the performance of different machine learning models for offensive language detection with and without the provided contextual information. The results showed that while the contextual information slightly increased the macro-averaged F1-score of certain models, it generally did not significantly improve the performance of the models.\n\nAreas of improvement: \n-  Given the limited impact of context in this study, it might be interesting to explore other forms of context, such as user profiles, user history, or even the broader conversation thread.\n- Bias Analysis: A deeper dive into potential biases introduced by the choice of target group and how they might be mitigated in future studies would be beneficial.\n", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "None"}, "extra_text": {}}, {"review_id": "3R3SNO36o4", "reviewer": "Reviewer_xKL1", "summary": "", "questions": "Could you add full list of keywords? (since it's only 16 of them)", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. We briefly address the comments and questions be-low.\n\n**Q:** “Without full list of keywords (which contains just 16 of them, so could be easily pro-vided), it is difficult to understand what kind of data is being extracted in Tweets.”\n\n**A:** The keywords are phrases having the meaning “aşısız” (unvaccinated) with singular-plural form or spellings due to the Turkish character related issues. Here is the full list of keywords, these will also be included to the Appendix of the paper. We thank the reviewer for pointing this.\n\naşısız\n\nasısız\n  \naşısızlar \n\nasısızlar \n\naşı olmayan \n\nası olmayan \n\naşı olmayanlar \n\nası olmayanlar \n\naşı olmak istemeyen \n\nası olmak istemeyen \n\naşı olmak istemeyenler \n\nası olmak istemeyenler \n\naşı yaptırmayan \n\nası yaptırmayan \n\naşı yaptırmayanlar \n\nası yaptırmayanlar\n\n\n**Q:** “It seems like the authors were the annotators of the dataset. This could be fine for train-ing set, but inappropriate for test set.”\n\n**A:** The main contribution of the paper is the annotated data set and the annotation process was time consuming and laborious. Therefore, the annotators were included as authors to the paper due to their significant annotation efforts. A general guideline for annotation was determined and provided to the annotators and a training was performed by using sample examples. As explained in the Dataset sec-tion, each tweet was annotated by two annotators independently, a third annotator was used to resolve inconsistencies and tweet pairs with ambiguity were filtered by following the procedure described in the Dataset section of the paper. The inter-annotator agreement F1 score and Cohen Kappa score for the final version of the dataset was calculated as 95.21% and 88.97%, respectively.  \nThe models used in the paper are standard models with the main goal of providing a baseline for in-vestigating the impact of context. Feature engineering was not performed; thus, there was no any an-notator input injected into the models. \n\n**Q:** “Results are hard to evaluate (and to replicate). No details about the fine-tuning of the models were provided. Also, it is unclear how the data was fed to the transformer (this may change how we can interpret the results)”\n\n**A:** We thank the reviewer for this comment. We will include these details to the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Turkish is a low resource language. Hence, having more human annotated datasets can potentially be valuable.\n2. Reporting negative results can potentially be helpful for other scientists.", "reasons_to_reject": "The paper has significant amount of scientific errors as well as missing to convey the significance of the results.\n\n1. Without full list of keywords (which contains just 16 of them, so could be easily provided), it is difficult to understand what kind of data is being extracted in Tweets.\n2. It seems like the authors were the annotators of the dataset. This could be fine for training set, but inappropriate for test set. The papers states that the authors randomly selected 20% of the self-annotated dataset. This is problematic. Tests set should be a separate, never seen, independently annotated set.\n3. It appears that the third author was used as an adjudicator for situation where the other two annotators disagreed. It would be more appropriate ti have an independent, trained adjudicator.\n4. Results are hard to evaluate (and to replicate). No details about the fine-tuning of the models were provided. Also, it is unclear how the data was fed to the transformer (this may change how we can interpret the results)\n5. Overall scientific value of the paper is low (since the annotation and the adjudication was done by the authors).", "questions_to_authors": "Could you add full list of keywords? (since it's only 16 of them)", "paper_topic_and_main_contributions": "The paper presents a Turkish dataset of pairs of tweets where one tweet is a reply to the other one. The dataset was manually annotated with binary labels (for \"offensive language against unvaccinated people\" or not) by the authors. The paper also reports the performance of classifiers based on the basic machine learning models and the collected dataset. This papers reports negative results.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The language of the paper is rather informal. Some parts of the paper are confusing for understanding (dataset section)."}, "extra_text": {}}, {"review_id": "9P1WNHF844", "reviewer": "Reviewer_tdPF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments and feedback to further improve the paper. We briefly address the comments and questions below.\n\n**Q:** “Some decisions made for dataset construction don’t seem reasonable/justified to me. For example, the authors limited the definition to offensive/non-offensive labels to a certain aspect of a topic (against covid vaccinated people) (lines 159 – 168) and even if a tweet is offensive but not fitting this criterion, it was labelled as non-offensive. I think this limited the scope and ap-plicability of the dataset greatly, especially with the COVID-19 topic becoming almost obsolete these days.”\n\n**A:** Our aim was to create a target-specific data set, where the target group for offensive language is unvaccinated people in the Covid-19 pandemic. So, rather than detecting offensive language, our goal was to enable the development of models that are able to detect offensive language towards a certain target, which is a more challenging task than detecting the existence of offensive language in general. We will explain this more clearly in the paper. \nWe chose the Covid-19 topic, because we believe infectious diseases and vaccination/unvaccination will continue to be important topics and the created dataset may be useful for further analysis in other disciplines as well, such as social or health sciences. \n\n**Q:** “Due to the extreme skewness of the dataset (only 5% of the dataset has a positive label), this means the way the test set was constructed (randomly sample), it will be easy to defeat by a majority label baseline.”\n\n**A:** We thank the reviewer for this comment. We will report the performance of the majority baseline in the revised paper. Such a baseline will not be able to identify the positive class (minority class). We will also report the performance separately for each class (positive and negative) to demonstrate the success of the classifiers for both the majority and the minority classes.\n\n**Q:** “Sec 2 could have really benefited from real tweet examples from the dataset.”\n\n**A:** We have sample tweet pairs in the Appendix, but we will also include examples in Section 2 to en-hance the definitions and descriptions. We thank the reviewer for this suggestion.\n\n**Q:** \"Dataset 3 and the other two isn’t clear. How can a reply tweet not reply to any other tweet?\"\n\n**A:** It is not “a reply tweet” not reply to any other tweet, but instead pairs where “the replied tweet” contains these keywords, yet it is not a response to any other tweet. We agree that this explanation is confusing, and we will clarify this with an example in the revised paper. In Dataset 2, the “replied” (that is original) tweet contains these keywords, and that tweet itself is a reply to another tweet. In Dataset 3, that tweet is not a reply to another tweet. Suppose we have three tweets T1, T2 and T3 and suppose, T2 is a reply to T1 and T3 is a reply to T2 (i.e.,  T1 > T2 > T3). In that case, T2 is included to Dataset 2, whereas T1 to Dataset 3 if they contain at least one of the keywords. Note that T1 is not a response to any other tweet, it is just a replied tweet. On the other hand, T2 is a reply tweet to T1, but replied tweet for T3. \n\n**Q:** \"there are several types of tweets (reply tweets, replied tweets), please define each clearly.\"\n\n**A:** We thank the reviewer for pointing this issue, we will define each in the revised paper as follows:\n- reply tweet: A tweet which is written as an answer to another tweet\n- replied tweet: A tweet which was answered by another tweet\n\n**Q:** \"Since in case of disagreement between the first two annotators, these pairs were dropped (Except for ambiguous), then it is normal to have such high agreement compared to stage 1.\"\n\n**A:** Yes, in fact, our goal was obtaining a dataset with a reliable level of consistency; therefore, we fol-lowed such a strategy.\n\n**Q:** \"Also, providing stats on labels for every annotation stage would have been useful.\"\n\n**A:** We thank the reviewer for this comment. We will include these statistics to the revised paper.\n\n**Justification For Ethical Concerns:** We thank the reviewer for pointing this issue and we will dis-cuss this in the Ethics Statement section of the paper. Unfortunately, we do not know any alternative way to develop a public dataset related to offensive language detection in tweets. Due to the terms of use of Twitter we are not allowed to publish the tweets themselves, but only the tweet IDs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-\tThe developed dataset offers annotated tweets with contextual information which is different from existing Turkish datasets. The data is also large in scale.", "reasons_to_reject": "-\tSome decisions made for dataset construction don’t seem reasonable/justified to me. For example, the authors limited the definition to offensive/non-offensive labels to a certain aspect of a topic (against covid vaccinated people) (lines 159 – 168) and even if a tweet is offensive but not fitting this criterion, it was labelled as non-offensive. I think this limited the scope and applicability of the dataset greatly, especially with the COVID-19 topic becoming almost obsolete these days. Such dataset can also be very confusing to machine learning models, where the same language (e.g., offensive language is usually associated with common phrases) can appear in both offensive and non-offensive tweets (Since offensiveness is topic-dependent according to the data annotation guidelines). \n-\tDue to the extreme skewness of the dataset (only 5% of the dataset has a positive label), this means the way the test set was constructed (randomly sample), it will be easy to defeat by a majority label baseline. Such model should have been reported in the paper. Moreover, this raises the question on how much we can learn about the performance of the provided models on such test set. Such data skewness can also be relevant to the first weakness I reported.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper describes the construction of a dataset of 20K Turkish tweet-reply pairs manually annotated by offensive language presence. The problem of offensive language detection is also tackled by comparing the performance of different machine learning models with and without contextual information.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "-\tSec 2 could have really benefited from real tweet examples from the dataset.\n-\tIn Sec 2, there are several types of tweets (reply tweets, replied tweets), please define each clearly. \n-\tThe difference between Dataset 3 and the other two isn’t clear. How can a reply tweet not reply to any other tweet? “replied tweet contains these keywords, yet it is not a response to any other tweet”\n-\tLines 240-243: Since in case of disagreement between the first two annotators, these pairs were dropped (Except for ambiguous), then it is normal to have such high agreement compared to stage 1. Also, providing stats on labels for every annotation stage would have been useful.", "Justification_for_Ethical_Concerns": "The release of such data where individual tweets are labelled by a negative label (i.e., offensive language) can be ethically questionable if the identity of the tweet author is known. The paper stated that the tweet IDs will be released which means tweet author identity can be easily found. Authors didn't comment on this."}}, {"review_id": "Q0N535oBQ8", "reviewer": "Reviewer_UEvi", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments. We briefly address the comments and questions below.\n\n**Q:** “Positive class is less than 5% of the total dataset. Making the usability of the resource limited.”\n\n**A:** This is correct, yet we collected as much data as we could and the nature of the dataset turned out to have this skewness. Nevertheless, there are over 1500 tweets in the positive class and we believe the dataset would be a useful resource for further studies, not only for NLP, but also for social and health sciences.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Dataset in a low-resource language\n- Decent manual curation results", "reasons_to_reject": "- Positive class is less than 5% of the total dataset. Making the usability of the resource limited.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this work, the authors developed a Turkish dataset for for investigating the impact of context for offensive language\ndetection in tweets. The dataset creation is well documented and is a valuable resource as there aren't many datasets for this domain in the Turkish language. The dataset task is well evaluated with traditional methodologies. ", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This study proposes a dataset, which comprises of 20K+ tweet-reply pairs. To enhance contextual understanding, each reply incorporates the original tweet to which it was posted in response. Human annotators carefully labeled the dataset. This study also provides experimental results with and without contextual information. 

Among reviewers there is a consensus about the dataset as it provided replies as context. However, there are concerns about the annotation process, utilization of context and so on. 

Reviewer xHbP raised concern about the benefit of contextual information. Reviewer tdPF questioned about the skewed distribution issue in the dataset. While authors addressed the concern, however, reviewer still has concerns such as data distribution and data sharing, which reflected in the rebuttal.  Reviewer UEvi also raised the same concern regarding the data distribution.
Decision: Accept-Findings