Package: corpushash
Thecorpushashlibrary enables the performance of common NLP tasks on
sensitive documents without disclosing their contents. This is done by
hashing every token in the corpus along with a salt.protocol1. the client — in her own secure environment — takes her classified
documents and does the linguistic pre-processing (removal of stopwords,
tokenization, etc.)2. the client then hashes the tokens: she creates random salts for every
different token and hashes the concatenation of token+salt. the client keeps a
dictionary of the salts used for every token.3. the client sends the hashed tokens to the analyst for linguistic processing.
as there is a biunivocal correspondence between every hashed token and
plaintext token, NLP can occur in the same way as with any plaintext corpus.4. once the NLP is over, the analyst sends the results to the client, who then
uses the dictionary to recover the plaintext tokens and thus the results’
meaning.the libraryThe library requires as input:a tokenizedcorpusas a nested list, whose elements are themselves
nested lists of the tokens of each document in the corpus.each list corresponds to a document structure: its chapters,
paragraphs, sentences. you decide how the nested list is to be
created or structured, as long as the input is a nested list with
strings as their bottom-most elements..corpus_path, a path to a directory where the output files are to
be stored.The output includes:a.jsonfile for every document in thecorpus, named sequentially as
positive integers, e.g., the first document being0.json, stored incorpus_path/public/$(timestamp-of-hash)/.two.jsondictionaries stored incorpus_path/private. they are
used to decode the.jsonfiles or the NLP results.installthe package demands python ⩾3, but has no external dependencies.using pip:pip3installcorpushashor from repository (most up-to-date):pip3installgit+https://github.com/NAMD/corpushash.gitor manually:gitclonehttps://github.com/NAMD/corpushash.gitcdcorpushashpython3setup.pyinstallusagethis will hash each word in the first four verses of thezen of pythonto the same .json document:importcorpushashaschexample_corpus=[[['Beautiful','is','better','than','ugly'],['Explicit','is','better','than','implicit'],['Simple','is','better','than','complex'],['Complex','is','better','than','complicated'],['Flat','is','better','than','nested']]]hashed_corpus=ch.CorpusHash(example_corpus,'output_directory')this will hash each word in the first four verses of the zen of python tofourdifferent .json documents, as if they were different documents:importcorpushashaschexample_corpus=[['Beautiful','is','better','than','ugly'],['Explicit','is','better','than','implicit'],['Simple','is','better','than','complex'],['Complex','is','better','than','complicated'],['Flat','is','better','than','nested']]hashed_corpus=ch.CorpusHash(example_corpus,'output_directory')so be careful when constructing your nested lists! check the tutorial atnotebooks/tutorial.ipynb.notesprobability of collision is extremely low (check thepreprint), but
still we check for them, so they are not an issue.hashing the tokens is not the same as encrypting them. as the same token
always maps to the same hash, the resulting hashed corpus is subject to
frequency analysis. even if a pre-processed text is almost uncomprehensible to
a reader (specially if stopwords are removed), there probably is still a
degree of trust in the analyst. she is usually someone who has no incentive
to attempt a decipherment of the text or someone who has a lesser (but by no
means inexisting) security clearance. this vulnerability will be investigated
in the future.memory complexity is estimated to be at most double the size of the biggest
document in the corpus.credits@odanoburu & @fccoelholicenseLGPL 3, check theLICENSE.mdfile for full content.