Package: toxic-comment-collection
Data Integration for Toxic Comment Classification: Making More Than 40 Datasets Easily Accessible in One Unified FormatThis repository contains the code of our paperData Integration for Toxic Comment Classification:
Making More Than 40 Datasets Easily Accessible in One Unified Formataccepted for publication at this year's ACL workshop on Online Abuse and Harms (WOAH).We present a collection of more than 40 datasets in the form of a software tool that automatizes downloading and processing of the data and presents them in a unified data format that also offers a mapping of compatible class labels.
Another advantage of that tool is that it gives an overview of properties of available datasets, such as different languages, platforms, and class labels to make it easier to select suitable training and test data for toxic comment classification.SetupClone this repository and install viapip install .git clone git@github.com:julian-risch/toxic-comment-collection.git
cd toxic-comment-collection
pip install .UsageDownload individual datasets with theget_dataset()method into tab-separated files. A list of datasets is at thebottom of this web page.from toxic_comment_collection import get_dataset
get_dataset('basile2019')It's as simple as that. You can now work with the dataset, for example, with pandasimport pandas as pd
df = pd.read_csv("./files/basile2019/basile2019en.csv", sep="\t")
df.head()id                                               text    labels
0   0  Hurray, saving us $$$ in so many ways @potus @...  ['hate']
1   1  Why would young fighting age men be the vast m...  ['hate']
2   2  @KamalaHarris Illegals Dump their Kids at the ...  ['hate']
3   3  NY Times: 'Nearly All White' States Pose 'an A...        []
4   4  Orban in Brussels: European leaders are ignori...        []Some datasets require Twitter API credentials to be downloaded.
Enter your Twitter API credentials in a fileapi_config.jsonwith the followoing format. You can store that in a directory of your choice. For this example, it's stored in./src/toxic_comment_collection/api_config.json.{"twitter":{"consumer_key":"","consumer_secret":"","access_token":"","access_token_secret":""}}After filling outapi_config.json, theget_dataset()method can use it as input:get_dataset('albadi2018', api_config_path='./src/toxic_comment_collection/api_config.json')
df = pd.read_csv("./files/albadi2018/albadi2018ar_train.csv", sep="\t")
df.head()id                                               text    labels
0   0  مؤسسة أرشيف المغرب تتسلم وثائق عن ذاكرة اليهود...  ['none']
1   1  مفتي السعودية حماس إرهابية وقتال اليهود حرام ش...  ['none']
2   2      أمراء ال سعود اليهود يخوضون حربا عن الصهيونيه  ['hate']
3   3  تحميل كتاب مقارنة الأديان: اليهودية تأليف أحمد...   ['none']
4   4  #هزه_ارضيه_في_جده\n\nهذه هيه الهزه الحقيقيه وت...  ['hate']All datasets can be downloaded automatically, which will take some time. To respect rate limits of the Twitter API, the program might sleep for several minutes and then continue automatically.from toxic_comment_collection import get_all_datasets
get_all_datasets(api_config_path='./src/toxic_comment_collection/api_config.json')After downloading all datasets, they can be combined into one large tab-separated file. To this end, the file./src/toxic_comment_collection/config.jsondefines the mappings of different labels to a common subset as described in ourpaper. You can download ithere, as it is part of this GitHub repository. The resulting combined file is stored in./files/combined.tsv. Note that we skip downloading all datasets in the following command assuming you have already downloaded them:get_all_datasets(config_path="./src/toxic_comment_collection/config.json", skip_download=True, api_config_path='./src/toxic_comment_collection/api_config.json')A summary of all downloaded datasets can be generated with thegenerate_statistics()method:from toxic_comment_collection import generate_statistics
generate_statistics('./files')It creates a file calledstatistics.txt:######################
# Overall Statistics #
######################

rows:       812094
file size:  241226068
labels:     
    indirect:                        13479
    none:                            471720
    offensive:                       66742
...CitationIf you use our work, please cite our paperData Integration for Toxic Comment Classification:
Making More Than 40 Datasets Easily Accessible in One Unified Formatthat has been published at the ACL'21 Workshop on Online Abuse and Harms (WOAH) as follows:@inproceedings{risch-etal-2021-toxic,
title = "Data Integration for Toxic Comment Classification: Making More Than 40 Datasets Easily Accessible in One Unified Format",
author = "Risch, Julian and Schmidt, Philipp and Krestel, Ralf",
booktitle = "Proceedings of the Workshop on Online Abuse and Harms (WOAH)",
month = aug,
year = "2021",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.woah-1.17",
doi = "10.18653/v1/2021.woah-1.17",
pages = "157--163",
}Add a New DatasetCreate a new file in thedatasetsfolder with the following naming scheme:<paper_author><paper_year><suffix (for duplicate file names)>.pyComplete the content of the file like in the following example:importosfrom.importdatasetfrom.importhelpersclassMubarak2017aljazeera(dataset.Dataset):name="mubarak2017aljazeera"url="http://alt.qcri.org/~hmubarak/offensive/AJCommentsClassification-CF.xlsx"hash="afa00e36ff5492c1bbdd42a0e4979886f40d00f1aa5517807a957e22fb517670"files=[{"name":"mubarak2017ar_aljazeera.csv","language":"ar","type":"training","platform":"twitter"}]comment="""Annotation Meaning0 NORMAL_LANGUAGE-1 OFFENSIVE_LANGUAGE-2 OBSCENE_LANGUAGE"""license=""" """Overwrite the methodsprocessandunify_rowfrom the parent class (dataset.Dataset) to implement unpack and process the downloaded files. You might use methods fromdatasets/helpers.pyThe resulting.csvshould have the following columns:id (added automatically)textlabelsAdd the newly created file and class todatasets/helpers.py. (import+get_datasets()method.)Run the following command and answer everything with "n" to updateconfig.json:python main.py --genconfigWe are happy to adopt your changes. Just create a pull request from your fork to this repository.List of Datasets Included in the Collection#StateNameClass1DoneAre They our Brothers? Analysis and Detection of Religious Hate Speech in the Arabic TwittersphereAlbadi20182DoneMultilingual and Multi-Aspect Hate Speech Analysis (Arabic)Ousidhoum20193DoneL-HSAB: A Levantine Twitter Dataset for Hate Speech and Abusive Languagemulki20194DoneAbusive Language Detection on Arabic Social Media (Twitter)Mubarak2017twitter5DoneAbusive Language Detection on Arabic Social Media (Al Jazeera)Mubarak2017aljazeera6Postponed (OneDrive)Dataset Construction for the Detection of Anti-Social Behaviour in Online Communication in Arabic7Postponed (Login Required)Datasets of Slovene and Croatian Moderated News Comments8tar.bz2 fileOffensive Language and Hate Speech Detection for Danish9DoneAutomated Hate Speech Detection and the Problem of Offensive LanguageDavidson201710DoneHate Speech Dataset from a White Supremacy ForumGibert201811DoneHateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on TwitterWaseem201612DoneDetecting Online Hate Speech Using Context Aware ModelsGao201813DoneAre You a Racist or Am I Seeing Things? Annotator Influence on Hate Speech Detection on TwitterWaseem201614DoneWhen Does a Compliment Become Sexist? Analysis and Classification of Ambivalent Sexism Using Twitter DataJha201715Password requiredOverview of the Task on Automatic Misogyny Identification at IberEval 2018 (English)16DoneCONAN - COunter NArratives through Nichesourcing: a Multilingual Dataset of Responses to Fight Online Hate Speech (English)Chung201917Not suitedCharacterizing and Detecting Hateful Users on Twitter18DoneA Benchmark Dataset for Learning to Intervene in Online Hate Speech (Gab)Qian201919DoneA Benchmark Dataset for Learning to Intervene in Online Hate Speech (Reddit)Qian201920DoneMultilingual and Multi-Aspect Hate Speech Analysis (English)Ousidhoum201921Postponed (includes pictures)Exploring Hate Speech Detection in Multimodal Publications22Uses OLID DatasetPredicting the Type and Target of Offensive Posts in Social Media23DoneSemEval-2019 Task 5: Multilingual Detection of Hate Speech AgainstImmigrants and Women in TwitterBasile201924DonePeer to Peer Hate: Hate Speech Instigators and Their TargetsElSherief201825DoneOverview of the HASOC track at FIRE 2019: Hate Speech and Offensive Content Identification in Indo-European LanguagesMandl2019en26DoneLarge Scale Crowdsourcing and Characterization of Twitter Abusive BehaviorFounta201827E-Mail requiredA Large Labeled Corpus for Online Harassment Research28DoneEx Machina: Personal Attacks Seen at Scale, Personal attacksWulczyn2017attack29DoneEx Machina: Personal Attacks Seen at Scale, ToxicityWulczyn2017toxic30DoneDetecting cyberbullying in online communities (World of Warcraft)31DoneDetecting cyberbullying in online communities (League of Legends)32E-Mail requiredA Qality Type-aware Annotated Corpus and Lexicon for Harassment ResearchRezvan201833DoneEx Machina: Personal Attacks Seen at Scale, Aggression and FriendlinessWulczyn2017aggressive34DoneCONAN - COunter NArratives through Nichesourcing: a Multilingual Dataset of Responses to Fight Online Hate Speech (English)Chung201935DoneMultilingual and Multi-Aspect Hate Speech Analysis (English)Ousidhoum201936DoneMeasuring the Reliability of Hate Speech Annotations:The Case of the European Refugee CrisisRoss201737DoneDetecting Offensive Statements Towards Foreigners in Social MediaBretschneider201738DoneOverview of the GermEval 2018 Shared Task on the Identification of Offensive LanguageWiegand201839DoneOverview of the HASOC track at FIRE 2019: Hate Speech and Offensive Content Identification in Indo-European LanguagesMandl2019ger40Website DownDeep Learning for User Comment Moderation, Flagged Comments41Website DownDeep Learning for User Comment Moderation, Flagged Comments42DoneOffensive Language Identification in GreekPitenis202043Application requiredAggression-annotated Corpus of Hindi-English Code-mixed Data44Application requiredAggression-annotated Corpus of Hindi-English Code-mixed Data45DoneDid you offend me? Classification of Offensive Tweets in Hinglish LanguageMathur201846Dataset not availableA Dataset of Hindi-English Code-Mixed Social Media Text for HateSpeech Detection47DoneOverview of the HASOC track at FIRE 2019: Hate Speech and Offensive Content Identification in Indo-European LanguagesMandl2019hind48DoneHate Speech Detection in the Indonesian Language: A Dataset and Preliminary StudyAlfina201849DoneMulti-Label Hate Speech and Abusive Language Detection in Indonesian TwitterIbrohim201950DoneA Dataset and Preliminaries Study for Abusive Language Detection in Indonesian Social MediaIbrohim201851DoneAn Italian Twitter Corpus of Hate Speech against ImmigrantsSanguinetti201852Application requiredOverview of the EVALITA 2018 Hate Speech Detection Task (Facebook)53Application requiredOverview of the EVALITA 2018 Hate Speech Detection Task (Twitter)54DoneCONAN - COunter NArratives through Nichesourcing: a Multilingual Dataset of Responses to Fight Online Hate Speech (English)Chung201955XML import requiredCreating a WhatsApp Dataset to Study Pre-teen Cyberbullying56Files not foundResults of the PolEval 2019 Shared Task 6:First Dataset and Open Shared Task for Automatic Cyberbullying Detection in Polish Twitter57DoneA Hierarchically-Labeled Portuguese Hate Speech DatasetFortuna201958ARFF file formatOffensive Comments in the Brazilian Web: A Dataset and Baseline Results59EncryptedDatasets of Slovene and Croatian Moderated News Comments60Application requiredOverview of MEX-A3T at IberEval 2018: Authorship and Aggressiveness Analysis in Mexican Spanish Tweets6162Data not foundhatEval, SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter (Spanish)63DoneA Corpus of Turkish Offensive Language on Social MediaColtekin201964DoneAggression-annotated corpus of hindi-english code-mixed dataKumar201865DonePredicting the Type and Target of Offensive Posts in Social MediaZampieri2019DisclaimerThis is a utility library that downloads and transforms publicly available datasets. We do not host or distribute these datasets, vouch for their quality or fairness, or claim that you have license to use them. It is your responsibility to determine whether you have permission to use the dataset under the dataset's license.If you wish to add, update or remove a dataset, please get in touch through a GitHub pull request.