Package: text-sensitivity
Sensitivity testing (fairness & robustness) for text machine learning modelsExtension oftext_explainabilityUses thegeneric architectureoftext_explainabilityto also include tests ofsafety(how safe it the model in production, i.e. types of inputs it can handle),robustness(how generalizable the model is in production, e.g. stability when adding typos, or the effect of adding random unrelated data) andfairness(if equal individuals are treated equally by the model, e.g. subgroup fairness on sex and nationality).© Marcel Robeer, 2021Quick tourSafety: test if your model is able to handle different data types.fromtext_sensitivityimportRandomAscii,RandomEmojis,combine_generators# Generate 10 strings with random ASCII charactersRandomAscii().generate_list(n=10)# Generate 5 strings with random ASCII characters and emojiscombine_generators(RandomAscii(),RandomEmojis()).generate_list(n=5)Robustness: if your model performs equally for different entities ...fromtext_sensitivityimportRandomAddress,RandomEmail# Random address of your current locale (default = 'nl')RandomAddress(sep=', ').generate_list(n=5)# Random e-mail addresses in Spanish ('es') and Portuguese ('pt'), and include from which country the e-mail isRandomEmail(languages=['es','pt']).generate_list(n=10,attributes=True)... and if it is robust under simple perturbations.fromtext_sensitivityimportcompare_accuracyfromtext_sensitivity.perturbationimportto_upper,add_typos# Is model accuracy equal when we change all sentences to uppercase?compare_accuracy(env,model,to_upper)# Is model accuracy equal when we add typos in words?compare_accuracy(env,model,add_typos)Fairness: see if performance is equal among subgroups.fromtext_sensitivityimportRandomName# Generate random Dutch ('nl') and Russian ('ru') names, both 'male' and 'female' (+ return attributes)RandomName(languages=['nl','ru'],sex=['male','female']).generate_list(n=10,attributes=True)InstallationSee theinstallationinstructions for an extended installation guide.MethodInstructionspipInstall fromPyPIviapip3 install text_sensitivity.LocalClone this repository and install viapip3 install -e .or locally runpython3 setup.py install.DocumentationFull documentation of the latest version is provided athttps://text-sensitivity.readthedocs.io/.Example usageSeeexample_usage.mdto see an example of how the package can be used, or run the lines inexample_usage.pyto do explore it interactively.Releasestext_sensitivityis officially released throughPyPI.SeeCHANGELOG.mdfor a full overview of the changes for each version.Citation@misc{text_sensitivity,title={Python package text\_sensitivity},author={Marcel Robeer},howpublished={\url{https://git.science.uu.nl/m.j.robeer/text_sensitivity}},year={2021}}MaintenanceContributorsMarcel Robeer(@m.j.robeer)Elize Herrewijnen(@e.herrewijnen)TodoTasks yet to be done:Word-level perturbationsAdd fairness-specific metrics:Counterfactual fairnessAdd expected behaviorRobustness: equal to prior prediction, or in some cases might expect that it deviatesFairness: may deviate from original predictionTestsAdd tests for perturbationsAdd tests for sensitivity testing schemesAdd visualization abilityCreditsEdward Ma.NLP Augmentation. 2019.Daniele Faraglia and other contributors.Faker. 2012.Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin and Sameer Singh.Beyond Accuracy: Behavioral Testing of NLP models with CheckList.Association for Computational Linguistics(ACL). 2020.