Package: piicatcher
PII Catcher for Databases and Data WarehousesOverviewPIICatcher is a scanner for PII and PHI information. It finds PII data in your databases and file systems
and tracks critical data. PIICatcher uses two techniques to detect PII:Match regular expressions with column namesMatch regular expressions and using NLP libraries to match sample data in columns.Read more in theblog poston both these strategies.PIICatcher isbatteries-includedwith a growing set of plugins to scan column metadata as well as metadata.
For example,piicatcher_spacyusesSpacyto detect
PII in column data.PIICatcher supports incremental scans and will only scan new or not-yet scanned columns. Incremental scans allow easy
scheduling of scans. It also provides powerful options to include or exclude schema and tables to manage compute resources.There are ingestion functions for bothDatahubandAmundsenwhich will tag columns
and tables with PII and the type of PII tags.ResourcesAWS Glue & Lake Formation Privilege Analyzerfor an example of how piicatcher is used in production.Two strategies to scan data warehousesQuick StartPIICatcher is available as a docker image or command-line application.InstallationDocker:alias piicatcher='docker run -v ${HOME}/.config/tokern:/config -u $(id -u ${USER}):$(id -g ${USER}) -it --add-host=host.docker.internal:host-gateway tokern/piicatcher:latest'Pypi:
# Install development libraries for compiling dependencies.
# On Amazon Linux
sudo yum install mysql-devel gcc gcc-devel python-develpython3 -m venv .env
source .env/bin/activate
pip install piicatcher

# Install Spacy plugin
pip install piicatcher_spacyCommand Line Usage# add a sqlite source
piicatcher catalog add-sqlite --name sqldb --path '/db/sqldb/test.db'

# run piicatcher on a sqlite db and print report to console
piicatcher detect --source-name sqldb
╭─────────────┬─────────────┬─────────────┬─────────────╮
│   schema    │    table    │   column    │   has_pii   │
├─────────────┼─────────────┼─────────────┼─────────────┤
│        main │    full_pii │           a │           1 │
│        main │    full_pii │           b │           1 │
│        main │      no_pii │           a │           0 │
│        main │      no_pii │           b │           0 │
│        main │ partial_pii │           a │           1 │
│        main │ partial_pii │           b │           0 │
╰─────────────┴─────────────┴─────────────┴─────────────╯API UsageCode Snippet:fromdbcat.apiimportopen_catalog,add_postgresql_sourcefrompiicatcher.apiimportscan_database# PIICatcher uses a catalog to store its state.# The easiest option is to use a sqlite memory database.# For production usage check, https://tokern.io/docs/data-catalogcatalog=open_catalog(app_dir='/tmp/.config/piicatcher',path=':memory:',secret='my_secret')withcatalog.managed_session:# Add a postgresql sourcesource=add_postgresql_source(catalog=catalog,name="pg_db",uri="127.0.0.1",username="piiuser",password="p11secret",database="piidb")output=scan_database(catalog=catalog,source=source)print(output)# Example Output[['public','sample','gender','PiiTypes.GENDER'],['public','sample','maiden_name','PiiTypes.PERSON'],['public','sample','lname','PiiTypes.PERSON'],['public','sample','fname','PiiTypes.PERSON'],['public','sample','address','PiiTypes.ADDRESS'],['public','sample','city','PiiTypes.ADDRESS'],['public','sample','state','PiiTypes.ADDRESS'],['public','sample','email','PiiTypes.EMAIL']]PluginsPIICatcher can be extended by creating new detectors. PIICatcher supports two scanning techniques:MetadataDataPlugins can be created for either of these two techniques. Plugins are then registered using an API or usingPython Entry Points.To create a new detector, simply create a new class that inherits fromMetadataDetectororDatumDetector.In the new class, define a functiondetectthat will return aPIITypeIf you are detecting a new PII type, then you can define a new class that inherits from PIIType.For detailed documentation, checkpiicatcher plugin docs.Supported DatabasesPIICatcher supports the following databases:Sqlite3v3.24.0 or greaterMySQL5.6 or greaterPostgreSQL9.4 or greaterAWS RedshiftAWS AthenaSnowflakeBigQueryDocumentationFor advanced usage refer documentationPIICatcher Documentation.SurveyPlease take thissurveyif you are a user or considering using PIICatcher.
The responses will help to prioritize improvements to the project.Stats CollectionWe use cookies to a analyse our traffic and features usage.
We may share information about your use of our product for our social media and marketing purposes.
These cookies don't collect your sensitive and/or confidential information.
If you would like to opt out of these cookies, runpiicatcher--disable-statsTo Enable:piicatcher--enable-statsContributingFor Contribution guidelines,PIICatcher Developer documentation.