Package: pilsner
pilsnerPython implemented library servicing named entity recognition1. PurposeThis library is Python implementation of toolkit for dictionary based named
entity recognition. It is intended to store any thesaurus in a trie-like
structure and identify any of stored synonyms in a string.2. Installation and dependenciespipinstallpilsnerpilsneris tested in Python 3.6, 3.7, and 3.8.The only dependency issicpackage. While it can be automatically installed
at the time ofpilsnerinstallation, manual installation ofsicbeforehand
might also be considered (see benchmark of cythonized vs pure Python
implementation insicdocimentation,https://pypi.org/project/sic/).3. Diagrampilsnerconsists of two major components:ModelandUtility.Modelclass provides storage for the dictionary and string normalization rules, as
well as low-level methods for populating this storage.Utilityclass provides
high-level methods for storing and retrieving data to/fromModelinstance.4. Usageimportpilsner4.1. Initialize modelTo initialize empty model:m=pilsner.Model()To specify path to temporary database for empty model:m=pilsner.Model(storage_location='path/to/database.file')To create empty model that uses database created in memory rather than on
disk:m=pilsner.Model(storage_location=':memory:')To create empty model that does not store any attributes in a database at all:m=pilsner.Model(simple=True)If database is created in memory, the model cannot be later saved on disk
(can only be used instantly).To load model from disk:m=pilsner.Model(filename='path/to/model')More on how model is saved to and loaded from disk - see4.6. Save modeland4.7. Load model.4.2. Add string normalization unitsDepending on the dictionary and nature of the text supposed to be parsed,
string normalization might not be required at all, and nothing specific is to
be done here in such case.Without string normalization, synonyms from the dictionary will be stored as
they are and looked up by recognizer case-sensitively.To add a single normalization unit:# Assuming m is pilsner.Model instance:m.add_normalizer(normalizer_name='normalizer_tag',filename='path/to/normalizer_config.xml')String normalization is technically done bysiccomponent. See
documentation forsicathttps://pypi.org/project/sic/to learn how
to design normalizer config.Model can embed more than one normalization unit.Default normalization unit for the model is the one added first or the last
one added with parameterdefaultset toTrue.Having multiple normalization units in one model makes perfect sense when the
stored dictionary contains synonyms of different nature that should be
normalized in different ways (for example, abbreviations probably should not
get normalized at all, while other synonyms might include tokens or punctuation
marks that should not affect entity recognition). For that purpose, Model class
includesnormalizer_mapdict that is supposed to map names of added
normalization units to values in specific field in a dictionary designating the
way a synonym should be normalized (tokenizer field, or tokenizer column):# Assuming m is pilsner.Model instance:m.normalizer_map={'synonym_type_1':'normalizer_1','synonym_type_2':'normalizer_2'}The snippet above instructspilsnerto normalize synonyms that havesynonym_type_1value intokenizercolumn withnormalizer_1normalization unit, and normalize synonyms that havesynonym_type_2value
intokenizercolumn withnormalizer_2normalization unit. For more about
fields in a dictionary, see4.4. Define dictionary.4.3. Initialize utilityTo load dictionary intoModelinstance, as well as to parse text, theUtilityinstance is required:r=pilsner.Utility()4.4. Define dictionarySource dictionary forpilsnermust be delimited text file.Along with the source dictionary, specifications of the columns (fields) must
be provided as list where each item corresponds to a column (from left to
right). Each item in this list must be a dict object with string keysname,include,delimiter,id_flag,normalizer_flag, andvalue_flag, so
that:field['name']is a string for column title;field['include']is a boolean that must be set toTruefor the column
to be included in the model, otherwiseFalse;field['delimiter']is a string that is supposed to split single cell into
list of values if the column holds concatenated lists rather than individual
values;field['id_flag]is a boolean that must be set toTrueif the column is
supposed to be used for grouping synonyms (generally, entity ID is such
column), otherwiseFalse;field['normalizer_flag']is a boolean that must be set toTrueif the
column holds indication on what normalization unit must be applied to this
particular synonym, otherwiseFalse;field['value_flag']is a boolean that must be set toTrueif the column
holds synonyms that are supposed to be looked up when parsing a text,
otherwiseFalse.If dictionary has a column flagged withnormalizer_flag, synonym in each
row will be normalized with string normalization unit which name is mapped on
value in this column usingpilsner.Model.normalizer_mapdict. If value is
not amongpilsner.Model.normalizer_mapkeys, default normalization unit
will be used.4.5. Compile modelTo store dictionary inModelinstance, methodcompile_modelofUtilityinstance must be called with the following required parameters:model: pointer to initilizedModelinstance;filename: string with path and filename of source dictionary;fields: dict object with definitions of columns (see4.4. Define dictionary);word_separator: string defining what is to be considered word separator
(generally, it should be whitespace);column_separator: string defining what is to be considered column
separator (e.g.\tfor tab-delimited file);column_enclosure: string defining what is to be stripped away from cell
after row has been split into columns (typically, it should be\nfor new
line character to be trimmed from the rightmost column).# Assuming m is pilsner.Model instance and r is pilsner.Utility instance:r.compile_model(model=m,filename='path/to/dictionary_in_a_text_file.txt',fields=fields,word_separator=' ',column_separator='\t',column_enclosure='\n')To review optional parameters, see comments in the code.4.6. Save modelIfModelinstance has compiled dictionary, and if database location for theModelinstance is not explicitly set to':memory:', the data such instance
is holding can be saved to disk:# Assuming m is pilsner.Model instancem.save('path/to/model_name')The snippet above will write the following files:path/to/model_name.attributes: database with attributes (fields from the
dictionary that are not synonyms) - will only be written ifModelinstance
is not created withsimple=Trueparameter;path/to/model_name.keywords: keywords used for disambiguation;path/to/model_name.normalizers: string normalization units;path/to/model_name.0.dictionary: trie with synonyms;path/to/model_name.<N>.dictionary: additional tries with synonyms (<N>being integer number of a trie) in case more than one trie was created (see
comments in the code -pilsner.Utility.compile_modelmethod,item_limitparameter).4.7. Load modelTo initialize newModelinstance using previously saved data:m=pilsner.Model(filename='path/to/model_name')Alternatively, data can be loaded to previously initializedModelinstance:m=pilsner.Model()m.load('path/to/model_name')In both cases, the program will look for the following files:path/to/model_name.attributes: database with attributes (fields from the
dictionary that are not synonyms) - if not found,Modelinstance will work
as if it is initialized withsimple=Trueparameter, meaning no attributes
other than primary IDs could be processed;path/to/model_name.keywords: keywords used for disambiguation;path/to/model_name.normalizers: string normalization units;path/to/model_name.<N>.dictionary: tries with synonyms (<N>being
integer).4.8. Parse stringTo parse a string without filtering out any synonyms and output all
attributes of spotted entities:# Assuming m is pilsner.Model instance, r is pilsner.Utility instance,# and text_to_parse is string to parseparsed=r.parse(model=m,source_string=text_to_parse)The output will be dict object where keys are tuples for location of spotted
entity in a string (begin, end) and values are dicts for attributes that are
associated with identified entity ({'attribute_name': {attribute_values}}).To ignore entity by its label rather than some of its attributes, compiled
model can be adjusted usingpilsnet.Utility.ignore_node()method:# Assuming m is pilsner.Model instance, r is pilsner.Utility instancer.ignore_node(model=m,label='irrelevant substring')# substring 'irrelevant substring' will not be found by pilsner.Utility.parse()# even if it is present in the modelFor details about optional parameters, see comments in the code -pilsner.Utility.parse()function.5. ExampleEverything written above is put together in example code,
see/misc/example/directory in the project's repository.