Package: presidio-evaluator
Presidio-researchThis package features data-science related tasks for developing new recognizers forPresidio.
It is used for the evaluation of the entire system,
as well as for evaluating specific PII recognizers or PII detection models.
In addition, it contains a fake data generator which creates fake sentences based on templates and fake PII.Who should use it?Anyone interested indeveloping or evaluating PII detection models, an existing Presidio instance or a Presidio PII recognizer.Anyone interested ingenerating new data based on previous datasets or sentence templates(e.g. to increase the coverage of entity values) for Named Entity Recognition models.Getting startedFrom PyPIcondacreate--namepresidiopython=3.9
condaactivatepresidio
pipinstallpresidio-evaluator# Download a spaCy model used by presidio-analyzerpython-mspacydownloaden_core_web_lgFrom sourceTo install the package:Clone the repoInstall all dependencies, preferably in a virtual environment:# Create conda env (optional)condacreate--namepresidiopython=3.9
condaactivatepresidio# Install package+dependenciespipinstallpoetry
poetryinstall# To install with all additional NER dependencies (e.g. Flair, Stanza, CRF), run:# poetry install --with ner# Download a spaCy model used by presidio-analyzerpython-mspacydownloaden_core_web_lg# Verify installationpytestNote that some dependencies (such as Flair and Stanza) are not automatically installed to reduce installation complexity.What's in this package?Fake data generatorfor PII recognizers and NER modelsData representation layerfor data generation, modeling and analysisMultipleModel/Recognizer evaluationfiles (e.g. for Spacy, Flair, CRF, Presidio API, Presidio Analyzer python package, specific Presidio recognizers)Training and modeling codefor multiple modelsHelper functions forresults analysis1. Data generationSeeData Generator READMEfor more details.The data generation process receives a file with templates, e.g.My name is {{name}}.
Then, it creates new synthetic sentences by sampling templates and PII values.
Furthermore, it tokenizes the data, creates tags (either IO/BIO/BILUO) and spans for the newly created samples.For information on data generation/augmentation, see the data generatorREADME.For an example for running the generation process, seethis notebook.For an understanding of the underlying fake PII data used, see thisexploratory data analysis notebook.Once data is generated, it could be split into train/test/validation sets
while ensuring that each template only exists in one set.
Seethis notebook for more details.2. Data representationIn order to standardize the process,
we use specific data objects that hold all the information needed for generating,
analyzing, modeling and evaluating data and models. Specifically,
seedata_objects.py.The standardized structure,List[InputSample]could be translated into different formats:CONLLfrompresidio_evaluatorimportInputSampledataset=InputSample.read_dataset_json("data/synth_dataset_v2.json")conll=InputSample.create_conll_dataset(dataset)conll.to_csv("dataset.csv",sep="\t")spaCy v3frompresidio_evaluatorimportInputSampledataset=InputSample.read_dataset_json("data/synth_dataset_v2.json")InputSample.create_spacy_dataset(dataset,output_path="dataset.spacy")Flairfrompresidio_evaluatorimportInputSampledataset=InputSample.read_dataset_json("data/synth_dataset_v2.json")flair=InputSample.create_flair_dataset(dataset)jsonfrompresidio_evaluatorimportInputSampledataset=InputSample.read_dataset_json("data/synth_dataset_v2.json")InputSample.to_json(dataset,output_file="dataset_json")3. PII models evaluationThe presidio-evaluator framework allows you to evaluate Presidio as a system, a NER model,
or a specific PII recognizer for precision and recall and error-analysis.Examples:Evaluate PresidioEvaluate spaCy modelsEvaluate Stanza modelsEvaluate CRF modelsEvaluate Flair modelsEvaluate Azure Text Analytics4. Training PII detection modelsCRFTo train a vanilla CRF on a new dataset, seethis notebook. To evaluate, seethis notebook.spaCyTo train a new spaCy model, first save the dataset in a spaCy format:# dataset is a List[InputSample]InputSample.create_spacy_dataset(dataset,output_path="dataset.spacy")To evaluate, seethis notebookFlairTo train Flair models, see thishelper classor this snippet:frompresidio_evaluator.modelsimportFlairTrainertrain_samples="data/generated_train.json"test_samples="data/generated_test.json"val_samples="data/generated_validation.json"trainer=FlairTrainer()trainer.create_flair_corpus(train_samples,test_samples,val_samples)corpus=trainer.read_corpus("")trainer.train(corpus)Note that the three json files are created usingInputSample.to_json.For more informationBlog post on NLP approaches to data anonymizationConference talk about leveraging Presidio and utilizing NLP approaches for data anonymizationContributingThis project welcomes contributions and suggestions.  Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visithttps://cla.opensource.microsoft.com.When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.This project has adopted theMicrosoft Open Source Code of Conduct.
For more information see theCode of Conduct FAQor
contactopencode@microsoft.comwith any additional questions or comments.Copyright notice:Fake Name Generator identities by theFake Name Generatorare licensed under aCreative Commons Attribution-Share Alike 3.0 United States License.
Fake Name Generator and the Fake Name Generator logo are trademarks of Corban Works, LLC.