Package: binonymizer
binonymizerBinonymizer is a tool in Python that aims at tagging personal data1in a parallel corpus.For example, for a input like:URL1  URL2  My name is Marta and my email is fake@email.com    Mi nombre es Marta y mi email es fake@email.comBinonymizer's output will be:URL1 URL2 My name is <entity class="PER">Marta</entity> and my email is <entity class="EMAIL">fake@email.com</entity> Mi nombre es <entity class="PER">Marta</entity> y mi email es <entity class="EMAIL">fake@email.com</entity>Detectable entity tipesCurrently, the Binonymizer is able to detect and tag the following types of entities:PER: person namesORG: organism and company namesEMAIL: email addressesPHONE: phone numbersADDRESS: addressesID: personal card IDs (such as spanish DNIs)MISC: other personal data, or when the type it's uncertainOTHER: otherInstallation & RequirementsBinonymizer works with Python 3.6, and can be installed withpip:python3.6 -m pip install binonymizerAfter installation, two  binary files (binonymizerandbinonymizer-lite) will be located in yourpython/installation/prefix/bin directory.Language-dependant packages and models are automatically downloaded and installed on runtime, if needed.Extra instructions for basqueIn case you plan to binonymize basque data, you need to downloadbinonymizerfromgithub, and run the following steps:cdbinonymizer
gitsubmodulesync
gitsubmoduleupdate--init--recursive--remotecdprompsit_python_bindings
python3.6setup.pyinstallPlease note that you need to have access to Prompsit's private repository.Contact usif you need further details.UsageBinonymizer can be run with:binonymizer[-h]--format{tmx,cols}[--tmp_dirTMP_DIR][-bBLOCK_SIZE][-pPROCESSES][-q][--debug][--logfileLOGFILE][-v]input[output]srclangtrglangParameterspositional arguments:input: File to be anonymized (See format below)output: File with anonymization annotations (default: standard output)srclang: Source language code of the inputtrglang: Target language code of the inputoptional arguments:-h, --help: show this help message and exitMandatory:--format {tmx,cols}: Input file format. Values: cols, tmx  ("cols" format: URL1 URL2 SOURCE_SENTENCE TARGET_SENTENCE [extra columns] tab-separated)Optional:--tmp_dir TMP_DIR: Temporary directory where creating the temporary files of this program (default: default system temp dir, defined by the environment variable TMPDIR in Unix)-b BLOCK_SIZE, --block_size BLOCK_SIZE: Sentence pairs per block (default: 10000)-p PROCESSES, --processes PROCESSES: Number of processes to use (default: all CPUs minus one)Logging:-q, --quiet: Silent logging mode (default: False)--debug: Debug logging mode (default: False)--logfile LOGFILE: Store log to a file (default: standard error output)-v, --version: show version of this script and exitExamplebinonymizercorpus.en-es.rawcorpus.en-es.anonenes--formatcols--tmp_dir/tmpdir-b50000-p31This will read the corpus "corpus.en-es.raw", which is in a column-based format, extracting personal data and writing the tagged output in "corpus.en-es.anon". Binonymizer will run in blocks of 50000 sentences, using 31 cores, and writing temporary files in /tmpdirLite versionAlthoughbinonymizermakes use of parallelization by distributing workload to the available cores, some users might prefer to implement their own parallelization strategies. For that reason, a single-thread version of the script is provided:binonymizer_lite. The usage is exactly the same as for the full version, but omitting the blocksize (-b) and processes (-p) parameter.TO DOFully support TMX input/outputAddress recognitionGPU supportAutomate Prompsit-python-bindings submodule ( git submodule update --remote ,  python3.6 setup.py install)1: See EC definition of "personal information":https://ec.europa.eu/info/law/law-topic/data-protection/reform/what-personal-data_en