Package: detoxify
🙊 DetoxifyToxic Comment Classification with ⚡ Pytorch Lightning and 🤗 TransformersNews & Updates22-10-2021: New improved multilingual model & standardised class namesUpdated themultilingualmodel weights used by Detoxify with a model trained on the translated data from the 2nd Jigsaw challenge (as well as the 1st). This model has also been trained to minimise bias and now returns the same categories as theunbiasedmodel. New best AUC score on the test set: 92.11 (89.71 before).All detoxify models now return consistent class names (e.g. "identity_attack" replaces "identity_hate" in theoriginalmodel to match theunbiasedclasses).03-09-2021: New improved unbiased modelUpdated theunbiasedmodel weights used by Detoxify with a model trained on both datasets from the first 2 Jigsaw challenges. New best score on the test set: 93.74 (93.64 before).15-02-2021: Detoxify featured in Scientific American!Our opinion piece"Can AI identify toxic online content?"is now live on Scientific American14-01-2021: Lightweight modelsAdded smaller models trained with Albert for theoriginalandunbiasedmodels! Can access these in the same way with detoxify usingoriginal-smallandunbiased-smallas inputs. Theoriginal-smallachieved a mean AUC score of 98.28 (98.64 before) and theunbiased-smallachieved a final score of 93.36 (93.64 before).DescriptionTrained models & code to predict toxic comments on 3 Jigsaw challenges: Toxic comment classification, Unintended Bias in Toxic comments, Multilingual toxic comment classification.Built byLaura HanuatUnitary, where we are working to stop harmful content online by interpreting visual content in context.Dependencies:For inference:🤗 Transformers⚡ Pytorch lightningFor training will also need:Kaggle API (to download data)ChallengeYearGoalOriginal Data SourceDetoxify Model NameTop Kaggle Leaderboard Score %Detoxify Score %Toxic Comment Classification Challenge2018build a multi-headed model that’s capable of detecting different types of of toxicity like threats, obscenity, insults, and identity-based hate.Wikipedia Commentsoriginal98.8698.64Jigsaw Unintended Bias in Toxicity Classification2019build a model that recognizes toxicity and minimizes this type of unintended bias with respect to mentions of identities. You'll be using a dataset labeled for identity mentions and optimizing a metric designed to measure unintended bias.Civil Commentsunbiased94.7393.74Jigsaw Multilingual Toxic Comment Classification2020build effective multilingual modelsWikipedia Comments + Civil Commentsmultilingual95.3692.11It is also noteworthy to mention that the top leadearboard scores have been achieved using model ensembles. The purpose of this library was to build something user-friendly and straightforward to use.Multilingual model language breakdownLanguage SubgroupSubgroup sizeSubgroup AUC Score %🇮🇹 it849489.18🇫🇷 fr1092089.61🇷🇺 ru1094889.81🇵🇹 pt1101291.00🇪🇸 es843892.74🇹🇷 tr1400097.19Limitations and ethical considerationsIf words that are associated with swearing, insults or profanity are present in a comment, it is likely that it will be classified as toxic, regardless of the tone or the intent of the author e.g. humorous/self-deprecating. This could present some biases towards already vulnerable minority groups.The intended use of this library is for research purposes, fine-tuning on carefully constructed datasets that reflect real world demographics  and/or to aid content moderators in flagging out harmful content quicker.Some useful resources about the risk of different biases in toxicity or hate speech detection are:The Risk of Racial Bias in Hate Speech DetectionAutomated Hate Speech Detection and the Problem of Offensive LanguageRacial Bias in Hate Speech and Abusive Language Detection DatasetsQuick predictionThemultilingualmodel has been trained on 7 different languages so it should only be tested on:english,french,spanish,italian,portuguese,turkishorrussian.# install detoxifypipinstalldetoxifyfromdetoxifyimportDetoxify# each model takes in either a string or a list of stringsresults=Detoxify('original').predict('example text')results=Detoxify('unbiased').predict(['example text 1','example text 2'])results=Detoxify('multilingual').predict(['example text','exemple de texte','texto de ejemplo','testo di esempio','texto de exemplo','örnek metin','пример текста'])# to specify the device the model will be allocated on (defaults to cpu), accepts any torch.device inputmodel=Detoxify('original',device='cuda')# optional to display results nicely (will need to pip install pandas)importpandasaspdprint(pd.DataFrame(results,index=input_text).round(5))For more details check the Prediction section.LabelsAll challenges have a toxicity label. The toxicity labels represent the aggregate ratings of up to 10 annotators according the following schema:Very Toxic(a very hateful, aggressive, or disrespectful comment that is very likely to make you leave a discussion or give up on sharing your perspective)Toxic(a rude, disrespectful, or unreasonable comment that is somewhat likely to make you leave a discussion or give up on sharing your perspective)Hard to SayNot ToxicMore information about the labelling schema can be foundhere.Toxic Comment Classification ChallengeThis challenge includes the following labels:toxicsevere_toxicobscenethreatinsultidentity_hateJigsaw Unintended Bias in Toxicity ClassificationThis challenge has 2 types of labels: the main toxicity labels and some additional identity labels that represent the identities mentioned in the comments.Only identities with more than 500 examples in the test set (combined public and private) are included during training as additional labels and in the evaluation calculation.toxicitysevere_toxicityobscenethreatinsultidentity_attacksexual_explicitIdentity labels used:malefemalehomosexual_gay_or_lesbianchristianjewishmuslimblackwhitepsychiatric_or_mental_illnessA complete list of all the identity labels available can be foundhere.Jigsaw Multilingual Toxic Comment ClassificationSince this challenge combines the data from the previous 2 challenges, it includes all labels from above, however the final evaluation is only on:toxicityHow to runFirst, install dependencies# clone projectgitclonehttps://github.com/unitaryai/detoxify# create virtual envpython3-mvenvtoxic-envsourcetoxic-env/bin/activate# install projectpipinstall-edetoxifycddetoxify# for trainingpipinstall-rrequirements.txtPredictionTrained models summary:Model nameTransformer typeData fromoriginalbert-base-uncasedToxic Comment Classification Challengeunbiasedroberta-baseUnintended Bias in Toxicity Classificationmultilingualxlm-roberta-baseMultilingual Toxic Comment ClassificationFor a quick prediction can run the example script on a comment directly or from a txt containing a list of comments.# load model via torch.hubpythonrun_prediction.py--input'example'--model_nameoriginal# load model from from checkpoint pathpythonrun_prediction.py--input'example'--from_ckpt_pathmodel_path# save results to a .csv filepythonrun_prediction.py--inputtest_set.txt--model_nameoriginal--save_toresults.csv# to see usagepythonrun_prediction.py--helpCheckpoints can be downloaded from the latest release or via the Pytorch hub API with the following names:toxic_bertunbiased_toxic_robertamultilingual_toxic_xlm_rmodel=torch.hub.load('unitaryai/detoxify','toxic_bert')Importing detoxify in python:fromdetoxifyimportDetoxifyresults=Detoxify('original').predict('some text')results=Detoxify('unbiased').predict(['example text 1','example text 2'])results=Detoxify('multilingual').predict(['example text','exemple de texte','texto de ejemplo','testo di esempio','texto de exemplo','örnek metin','пример текста'])# to display results nicelyimportpandasaspdprint(pd.DataFrame(results,index=input_text).round(5))TrainingIf you do not already have a Kaggle account:you need to create one to be able to download the datago to My Account and click on Create New API Token - this will download a kaggle.json filemake sure this file is located in ~/.kaggle# create data directorymkdirjigsaw_datacdjigsaw_data# download datakagglecompetitionsdownload-cjigsaw-toxic-comment-classification-challenge

kagglecompetitionsdownload-cjigsaw-unintended-bias-in-toxicity-classification

kagglecompetitionsdownload-cjigsaw-multilingual-toxic-comment-classificationStart TrainingToxic Comment Classification Challenge# combine test.csv and test_labels.csvpythonpreprocessing_utils.py--test_csvjigsaw_data/jigsaw-toxic-comment-classification-challenge/test.csv--update_test

pythontrain.py--configconfigs/Toxic_comment_classification_BERT.jsonUnintended Bias in Toxicicity Challengepythontrain.py--configconfigs/Unintended_bias_toxic_comment_classification_RoBERTa_combined.jsonMultilingual Toxic Comment ClassificationThe translated data (source 1source 2) can be downloaded from Kaggle in french, spanish, italian, portuguese, turkish, and russian (the languages available in the test set).# combine test.csv and test_labels.csvpythonpreprocessing_utils.py--test_csvjigsaw_data/jigsaw-multilingual-toxic-comment-classification/test.csv--update_test

pythontrain.py--configconfigs/Multilingual_toxic_comment_classification_XLMR.jsonMonitor progress with tensorboardtensorboard--logdir=./savedModel EvaluationToxic Comment Classification ChallengeThis challenge is evaluated on the mean AUC score of all the labels.pythonevaluate.py--checkpointsaved/lightning_logs/checkpoints/example_checkpoint.pth--test_csvtest.csvUnintended Bias in Toxicicity ChallengeThis challenge is evaluated on a novel bias metric that combines different AUC scores to balance overall performance. More information on this metrichere.pythonevaluate.py--checkpointsaved/lightning_logs/checkpoints/example_checkpoint.pth--test_csvtest.csv# to get the final bias metricpythonmodel_eval/compute_bias_metric.pyMultilingual Toxic Comment ClassificationThis challenge is evaluated on the AUC score of the main toxic label.pythonevaluate.py--checkpointsaved/lightning_logs/checkpoints/example_checkpoint.pth--test_csvtest.csvCitation@misc{Detoxify,
  title={Detoxify},
  author={Hanu, Laura and {Unitary team}},
  howpublished={Github. https://github.com/unitaryai/detoxify},
  year={2020}
}