Package: impfic-core
impfic-coreCore code base for common functionalitiesInstallingpipinstallimpfic-coreUsageDealing with output from different parsersThe Doc class ofimpfic-coreoffers a unified API to parsed document from different parsers (currently SpaCy and Trankit).importspacyfromtrankitimportPipelineimportimpfic_core.parse.docasparse_docspacy_nlp=spacy.load('en_core_web_lg')trankit_nlp=Pipeline('english')# First paragraph of Moby Dick, taken from Project Gutenberg (https://www.gutenberg.org/cache/epub/2701/pg2701-images.html)text="""Call me Ishmael. Some years ago—never mind how long precisely—having little or no money in my purse, and nothing particular to interest me on shore, I thought I would sail about a little and see the watery part of the world. It is a way I have of driving off the spleen and regulating the circulation. Whenever I find myself growing grim about the mouth; whenever it is a damp, drizzly November in my soul; whenever I find myself involuntarily pausing before coffin warehouses, and bringing up the rear of every funeral I meet; and especially whenever my hypos get such an upper hand of me, that it requires a strong moral principle to prevent me from deliberately stepping into the street, and methodically knocking people’s hats off—then, I account it high time to get to sea as soon as I can. This is my substitute for pistol and ball. With a philosophical flourish Cato throws himself upon his sword; I quietly take to the ship. There is nothing surprising in this. If they but knew it, almost all men in their degree, some time or other, cherish very nearly the same feelings towards the ocean with me."""Documentobjects have the following properties:text(the whole text string)sentences,tokens,entitiesand optionalmetadata(a dictionary with whatever keys and values).# parse with both SpaCy and Trankitspacy_doc=spacy_nlp(text)trankit_doc=trankit_nlp(text)# First, turn SpaCy document object to an impfic Docimpfic_doc1=parse_doc.spacy_json_to_doc(spacy_doc.to_json())# Next, turn Trankit document object to an impfic Docimpfic_doc2=parse_doc.trankit_json_to_doc(trankit_doc)# Show type and length of impfic_core Doc# Doc length is number of tokensprint('impfic Doc of SpaCy parse:',type(impfic_doc1),len(impfic_doc1))print('impfic Doc of Trankit parse:',type(impfic_doc2),len(impfic_doc2))Outputs:>>>impficDocofSpaCyparse:<class'impfic_core.parse.doc.Doc'> 190>>>impficDocofTrankitparse:<class'impfic_core.parse.doc.Doc'> 226Sentenceobjects have the following properties:id: ID of the sentence in the document (running numbers)tokens: a list ofTokenobjectsentitites: a list ofEntityobjects (named entities identified by the parser)text: the sentence as text stringstart: the character offset of the start of the sentence within the documentend: the character offset of the end of the sentence within the documentExtracting Clausal Unitssent=doc.sentences[5]forsentindoc.sentences:print(sent.text)clauses=pattern.get_verb_clauses(sent)forclauseinclauses:print([t.textfortinclause])WithaphilosophicalflourishCatothrowshimselfuponhissword;Iquietlytaketotheship.clause:['With','a','philosophical','flourish','Cato','throws','himself','upon','his','sword',';','.']clause:['I','quietly','take','to','the','ship']External ResourcesTo use utilities for external resources such as the RBN, you need to point to your copy of those resources
in the settings (settings.py). Once you have done that, you can use them with:fromsettingsimportrbn_filefromimpfic_core.resources.rbnimportRBNrbn=RBN(rbn_file)rbn.has_term('aanbiddelijk')# returns TrueAnonymisationFor review anonymisation you need a salt hash in a file calledimpfic_core/secrets.py. The repository doesn't contain this file to ensure other cannot recreate the user ID mapping.
An example file is available asimpfic_core/secrets_example.py. Copy this file toimpfic_core/secrets.pyand update the salt hash to do your own user ID mapping.