Package: sanityze
sanityzeData scientists often need to remove or redact Personal Identifiable Information (PII) from their data. This package provides utilities to spot and redact PII from Pandas data frames.PII can be used to uniquely identify a person. This includes names, addresses, credit card numbers, phone numbers, email addresses, and social security numbers, and therefore regulatory bodies such as the European Union's General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA) require that PII be removed or redacted from data sets before they are shared an further processed.Contributors and MaintainersTony ZoghtCaesar WongJonah HamiltonWhysanityze?Because it's a fun name and it's a play on the word "sanitize" which is what we are doing to the data.Similar packages in PythonThe closet Python package in functionality to sanityze isscrubadubwhich is a package for finding and removing PII from text. The package is not designed to work with Pandas data frames, or other data structures, and we believe that our package will be more useful to data scientists, as we add more spotters (mechanisms for finding PII), support for more data structures, and provide mechanisms for users to define their own spotters.Quick StartTo get started withsanityze, install it usingpip:pipinstallsanityzeAnd visit thedocumentationfor more information and examples.Features and UsageConceptually,sanityzeis a package that provides a way to remove PII from Pandas data frames. The package provides a number of default spotters, which can be used to identify PII in the data and redact them.The main entry point to the package is theCleanserclass. TheCleanserclass is used to addSpotters to the cleanser, which will be used to identify PII in the data. The cleanser can then be used to cleanse the data, and redact the PII from the given data frame (all future data structures that will be suppportd by the package, in the future).The package comes with a number of default spotters, as subclassess ofSpotter:CreditCardSpotter- identifies credit card numbersEmailSpotter- identifies email addressesSpotters can be added to it using theadd_spotter()method. The cleanser can then be used to cleanse data using thecleanse()method which takes a Pandas data frame and returns a Pandas data frame with PII redacted.The redaction options provided by `sanityze`` are:Redact using a fixed string - The string in this case is the ID of the spotter. For example, if the spotter is an instance ofCreditCardSpotter, the string will be{{CREDITCARD}}, or{{EMAILADDRS}}for an instance ofEmailSpotter.Redact using a hash of the input - The hash is computed using thehashlibpackage, and the hash function ismd5. For example, if the spotter is an instance ofCreditCardSpotter, the string will be{{6a8b8c6c8c62bc939a11f36089ac75dd}}, if the input is contains a PII1234-5678-9012-3456.Classes and FunctionsCleanser: the main class of the package. It is used to add spotters to it, and then cleanse data using the spotters.add_spotter(): adds a spotter to the cleanserremove_spotter(): removes a spotter from the cleanserclean(): cleanses the data in the given data frame, and returns a new data frame with PII redactedEmailSpotter: a spotter that identifies email addressesgetUID(): returns the unique ID of the spotterprocess(): performs the PII matching and redactionCreditCardSpotter: a spotter that identifies credit card numbersgetUID(): returns the unique ID of the spotterprocess(): performs the PII matching and redactionYou can checkout detailed API Documentationshere.Below is a simple quick start example:importpandasaspdfromsanityzeimportCleanser,EmailSpotter# Create a cleanser, and don't add the default spotterscleanser=Cleanser(include_default_spotters=False)cleaner.add_spotter(fromsanityzeimportCleanser,EmailSpotter())cleaned_df=cleanser.clean(df)High-level DesignTo better understand the design of the package, we have provided a high-level design document, which will be kept up to date as the package evolves. The document can be foundhere.ContributingInterested in contributing? Check out thecontributing guidelines. Please note that this project is released with a Code of Conduct. By contributing to this project, you agree to abide by its terms.Licensesanityzewas created by Caesar Wong, Jonah Hamilton and Tony Zoght. It is licensed under the terms of theMIT license.Creditssanityzewas created withcookiecutterand thepy-pkgs-cookiecuttertemplate.Quick LinksPyPIRead the DocsDocumentation on GHKanban BoardIssuesHigh Level DesignContributing GuidelinesCode of ConductLicense