Package: hatecomp
Hate Datasets CompilationContains 17 different hate, violence, and discrimination speech datasets, along with anotations on where they were found, the data format and method for collection and labeling. Each dataset is kept in the original file structure and placed inside the respective folder'sdatafile, in case a more recent version is obtained. (Links for the dataset's source are in each ABOUT.md file)DatasetsAdditional information about each dataset can be found in the corresponding ABOUT.md file.The following Datasets are implemented:ZeerakTalat NAACLZeerakTalat NLPCSSHASOCVicomtechTwitterSexismMLMAWikiToxicityWikiAggressionWikiPersonalAttacksSeveral more have downloaders already, and are close to completionNotesTwo of the datasets,MLMA DatasetandOnline Intervention Dataset, only contain hateful posts, instead labeling other features such as the target of hate. Models trained on these datasets may be biased because of this.InstallingTo use the hatecomp datasets or models, simply run the following command in a python environment of your choice:pipinstallhatecompIf you do not have pytorch already installed, it is recommended to do so using conda. Visit the pytorchwebsitefor more information.Once it has finished downloading, you can start loading in datasets and models. Below are a couple of examples to get you started. For more advanced usage, please seescripts/train.py.ExamplesHere are a couple examples of how to use the hatecomp library.Working with a DatasetLoading datasets is very simple. Each has its own downloading script that will run lazily when you try to create the dataset. If you would like to, you can specify where to download and if the dataset should download. By default the datasets only download when they cannot find the necessary files in the given location.fromhatecomp.datasetsimportVicomtech# load a dataset from the default location,# or download the dataset in the default locationdataset=Vicomtech()example=dataset[0]# load a dataset from a specified location,# or download to that locationdataset=Vicomtech(root="my/special/dataset/path")example=dataset[0]# only load a dataset if it can be found at the given locationdataset=Vicomtech(root="my/special/dataset/path",download=False)example=dataset[0]The datasets also come equipped with a couple of handy features designed especially for NLP use and convenience.fromhatecomp.datasetsimportVicomtech# Mapping a function over the dataset data (usually text, unless the dataset has already been mapped)# Note that the map function can support batching if your mapped function supports it.defmy_tokenizing_function(some_string):return0dataset=Vicomtech()tokenized_dataset=dataset.map(function=my_tokenizing_function,batched=False)# Splitting the datasettrain_split,test_split=tokenized_dataset.split(test_proportion=0.1)Using a modelHatecomp also provides functionality for training models with these datasets, along with some pretrained models.Importing a pretrained modelLoading one of our pretrained models is quite simple, and only requires using the name of the appropriate dataset to do so. The model will then be downloaded into the files of the localhatecomppackage. Note this means that uninstallinghatecompwill delete the models, and this is intended behavior.fromhatecomp.modelsimportHatecompClassifier,HatecompTokenizer# To load an already downloaded modelmodel=HatecompClassifier.from_hatecomp_pretrained("Vicomtech")tokenizer=HatecompTokenizer.from_hatecomp_pretrained("Vicomtech")# To download a model if it does not exist locallymodel=HatecompClassifier.from_hatecomp_pretrained("Vicomtech",download=True)tokenizer=HatecompTokenizer.from_hatecomp_pretrained("Vicomtech")# Force download a model (useful if the files become corrupted for any reason)model=HatecompClassifier.from_hatecomp_pretrained("Vicomtech",force_download=True)tokenizer=HatecompTokenizer.from_hatecomp_pretrained("Vicomtech")The tokenizers also have the samedownloadandforce_downloadflags available, but if you are loading the tokenizer directly after the model, the files will be installed locally already, as the download retrieves the necessary data for both the model and the tokenizer.Training modelsThe process for training a model is quite simple, as there is a custom trainer class designed specifically for the datasets and models. Also included is a convenienceDataLoaderwrapper, which will handle collating the hatecomp data, since hatecomp datasets return ids, and the basetorch.utils.data.DataLoaderdoes not handle those by default.importtorchfromhatecomp.datasetsimportVicomtechfromhatecomp.datasets.baseimportDataLoaderfromhatecomp.trainingimportHatecompTrainerfromhatecomp.modelsimportHatecompClassifier,HatecompTokenizerdataset=Vicomtech()model=HatecompClassifier.from_huggingface_pretrained("roberta-base",dataset.num_classes)tokenizer=HatecompTokenizer.from_huggingface_pretrained("roberta-base",)train_set,test_set=dataset.split(0.1)train_dataloader=DataLoader(train_set,batch_size=32)test_dataloader=DataLoader(test_set,batch_size=64)optimizer=torch.optim.AdamW(model.parameters(),lr=3e-5)scheduler=torch.optim.lr_scheduler.OneCycleLR(optimizer,max_lr=3e-5,steps_per_epoch=len(train_dataloader),epochs=5)loss_function=torch.nn.functional.cross_entropytrainer=HatecompTrainer(root="root_directory",model=model,tokenizer=tokenizer,optimizer=optimizer,scheduler=scheduler,loss_function=loss_function,train_dataloader=train_dataloader,test_dataloader=test_dataloader,epochs=5)trainer.train("cuda")For an even more expedited training process, there is also theHatecompTrialRunner, an example of using this class can be found inscripts/hyperopt.py.ResultsHere is a list of results acheived on various datasets with Huggingface models, along with the SOTA performance (as best I could find). Since it is not always possible to find SOTA scores for obscure datsets measured with a particular metric, the hatecomp score is selected to match whatever SOTA could be found. (The links are locations where the SOTA reference was found. If you are looking for citations, please refer to theAbout.mdfor each dataset)DatasetMetricSOTAhatecomp/huggingfaceVicomtechAccuracy0.790.93ZeerakTalat-NAACLF10.740.94ZeerakTalat-NLPCSSF10.530.76HASOCF1 (Macro Average)0.530.55TwitterSexismF1 (Macro)0.870.99MLMAF1 (Multitask Macros EN)[0.30, 0.43,0.18,0.57][0.58,0.64, 0.16, 0.51](If you know of a better SOTA than what is listed here, please create an issue or pull request.)Also note that some of these datasets require tweet data. For these, a large number of tweet_ids return Unauthorized from the twitter API, so the data which the hatecomp models trained on is a subset of the total dataset. More information can be found in the following table:DatasetTotal SizeSuccessfully Downloaded TweetsAvailable Training PortionZeerakTalat-NAACL1690772100.4264ZeerakTalat-NLPCSS690953850.77941TwitterSexism1058350540.4775This info is valid as of Feb 2022, and is probably subject to change as Twitter continues to lock down their API.