Package: pii-extract-plg-transformers
Pii Extractor plugin: TransformersThis repository builds a Python package that installs a pii-extract-base
plugin to perform PII detection for text data using the Hugging Face
Transformers Python library. It will download and use trained token
classification models running on that library.The name of the plugin entry point ispiisa-detectors-transformersRequirementsThe package needsat least Python 3.8the pii-data and the pii-extract-base base packagesthe Transformers packagePyTorch, either GPU or CPUan NLP engine model for the desired language (will be downloaded on demand,
based on the configuration)InstallationInstall the package:pip install pii-extract-plg-transformers(it will
automatically install its dependencies,exceptPytorch)Install PyTorch: either the CPU PyTorch package or the GPU package
appropriate for your GPUIf necessary, define the cache directory for models (see
below)Cache directoryThe transformers library downloads models on the fly from the HuggingFace
Hub. It keeps them in a cache in a local folder, to avoid repeated downloads.Thepii-extract-plg-transformerspackage defines this local folder as
follows:If theHUGGINGFACE_HUB_CACHEenvironment variable is defined, it is
usedElse, if the configuration file for the package contains acachedirfield
inside thetask_configsection, it will be usedIf that field contains afalsevalue, thennospecific cache directory
will be defined (so the HuggingFace internal default will be used)Else, a default is chosen: thevar/piisa/hf-cachesubfolder in
the virtualenv that holds the packageUsageThe package does not have any user-facing entry points (except for two auxiliary
console scripts, see below). Instead, upon installation it
defines a plugin entry point. This plugin is automatically picked up by the
scripts and classes in pii-extract-base, and thus its functionality is exposed
to them.The task created by the plugin is a standard PII task object, using thepii_extract.build.task.MultiPiiTaskclass definition. It will be called,
as all PII task objects, with aDocumentChunkobject containing the data to
analyze. The chunkmustcontain language specification in its metadata, so
that the plugin knows which language to use (unless the plugin task has been
built withonly onelanguage; in that case if the chunk does not contain
a language specification, it will use that single language).ConfigurationRuntime behaviour is governed by a PIISA configuration file, which sets up which
models from the HuggingFace Hub will be downloaded and used (note that the
configuration defines the total set of languages available for detection, but
it is also possible to initialize the plugin with asubsetof the configuration
languages).The default configuration file defines detection forPersonandLocationPII instances for English, Spanish and French, using the WikiNEuRal multilingual
NER model available in the Hugging Face Hub.However, a configuration file can
also define a different model per language, and a different set of PII to detect
for each model (and also different aggregation strategies to merge the model
output). There is another example available.Auxiliary scriptsInformationpii-extract-transformers-infois a command-line script which provides
information about the plugin capabilities:version: installed package versionsmodels: list of configured Transdormers modelsmodel-entities: the total list of entities each configured model can
generatepii-entities: the PIISA tasks that this plugin will create, by translating
from the entities detected by the models (this depends on the PIISA config
used)Testingpii-extract-transformers-detectis a command-line script to do initial
testing: it performs PII detection by processing a text chunk through one of the
models defined in the plugin configuration.Note that this script instantiates the plugin task directly, i.e. it doesnotgo through the standard PIISA software stack (which would execute the task via
plugin loading into the pii-extract framework). For the same reason, itonlyexecutes this detection task, ignoring any other pii-extract plugins that
might be available.BuildingThe provided Makefile can be used to process the package:make pkgwill build the Python package, creating a file that can be
installed withpipmake unitwill launch all unit tests (using pytest, so pytest must be
available)make installwill install the package in a Python virtualenv. The
virtualenv will be chosen as, in this order:the one defined in theVENVenvironment variable, if it is definedif there is a virtualenv activated in the shell, it will be usedotherwise, a default is chosen as/opt/venv/pii(it will be
created if it does not exist)