Package: sic
sic(Latin)so, thus, such, in such a way, in this way(English)spelling is correctsicis a module for string normalization. Given a string, it separates
sequences of alphabetical, numeric, and punctuation characters, as well
as performs more complex transformation (i.e. separates or replaces specific
words or individual symbols). It comes with set of default normalization rules
to transliterate and tokenize greek letters and replace accented characters
with their base form. It also allows using custom normalization configurations.Basic usage:importsicbuilder=sic.Builder()machine=builder.build_normalizer()x=machine.normalize('abc123xyzalphabetagammag')print(x)The output will be:abc123xyzalphabetagammagInstallationsicis designed to work in Python 3 environment.siconly needs Python Standard Library (no other packages).To get wheel for Windows (Python >= 3.6) or source code package for Linux:pipinstallsicTo get source code package regardless the OS:pipinstallsic--no-binarysicWheels and .tar.gz can also be downloaded from the project's repository.Wheels contain binaries compiled from cythonized code. Source code package is
pure Python. Cythonized version performs better on short strings, while
non-cythonized version performs better on long strings, so one may be preferred
over another depending on usage scenario. The benchmark is below.STRING LENGTHREPEATSVERSIONMEAN TIME (s)7110000.tar.gz1.87110000wheel0.57100001.tar.gz2.77100001wheel15.9Tokenization configssicimplements tokenization, i.e. it splits a given string into tokens and
processes those tokens according to the rules specified in a configuration
file. Basic tokenization includes separating groups of alphabetical, numerical,
and punctuation characters within a string, thus turning them into separate
words (for future reference, we'll call such wordstokens). For instance,abc-123will be transformed intoabc - 123, having tokensabc,-, and123.What happens next to initially tokenized string must be defined using XML in
configuration file(s). Entry point to default tokenizer applied to a string issic/tokenizer.standard.xml.Below is the template and description for tokenizer config.<!-- tokenizer.config.xml --><!--This is the description of config file for tokenizer.General structure:<tokenizer>+-<import>+-...+-<import>+-<setting>+-...+-<setting>+-<split>+-...+-<split>+-<token>+-...+-<token>+-<character>+-...+-<character>--><?xml version="1.0" encoding="UTF-8"?><!-- There must be single root element, and it must be <tokenizer>: --><tokenizername="$name"><!-- $name: string label for this tokenizer --><!--Direct children of <tokenizer> are <import>, <setting>, <split>,<token>, and/or <character> elements (there can be zero to manydeclarations of any of those)--><!-- <import> elements point at other tokenizer configs to merge with --><importfile="$file"/><!-- $file: path to file with another tokenizer config --><!-- <setting> elements define high-level tokenizer settings --><settingname="$name"value="$value"/><!--Names and value requirements for /tokenizer/setting elements:$name="cs": $value="0"|"1" (if "1", this tokenizer will be case-sensitive)$name="bypass": $value="0"|"1" (if "1", this tokenizer will do nothing,regardless the rest content of this file)--><!--<split> elements define substrings that should be separated from text astokens--><splitwhere="$where"value="$value"/><!--$where="l"|"r"|"m" ("l" for left, "r" for right, "m" for middle)$value: string that will be handled as token when it's either in thebeginning of word ($where="l"), at the end of word ($where="r"), or inthe middle ($where="m")--><!--<token> elements define tokens that should be replaced with other tokens(or with nothing => removed)--><tokento="$to"from="$from"/><!--$to: string that should replace the token specified in $from$from: string that is the token to be replaced by another string specifiedin $to--><!--<character> elements define single characters that should be replaced withother single characters--><characterto="$to"from="$from"/><!--$to: character that should replace the another character specified in $from$from: character that should be replaced by another character specified in$to--></tokenizer>Below are descriptions and examples of tokenizer config elements.ELEMENTATTRIBUTESDESCRIPTIONEXAMPLE<import>file="path/to/another/config.xml"Import tokenization rules from another tokenizer config.<setting>name="bypass" value="?"If present and value="1", all tokenization rules are ignored, as if there was no tokenization at all (left for debug purposes).<setting>name="cs" value="?"If value="1", string is processed case-sensitively; if value="0" - case-insensitively; if not present, tokenizer is case-insensitive.<split>where="l" value="?"Separates token specified invaluefromleftpart of a bigger token.where="l" value="kappa":nf kappab-->nf kappa b<split>where="m" value="?"Separates token specified invaluewhen it is found in themiddleof a bigger token.where="m" value="kappa":nfkappab-->nf kappa b<split>where="r" value="?"Separates token specified invaluefromrightpart of a bigger token.where="r" value="gamma":ifngamma-->ifn gamma<token>to="?" from="?"Replaces token specified infromwith another token specified into.to="protein" from="gene":nf kappa b gene-->nf kappa b protein<character>to="?" from="?"Replaces character specified infromwith another character specified into.to="e" from="ë":citroën-->citroenAttributewhereof<split>element may have any combination ofl,m, orrliterals if the specified substring is required to be separated in different
places of a bigger string. So, instead of three different elements<splitwhere="l"value="word"/><splitwhere="m"value="word"/><splitwhere="r"value="word"/>using the following single one<splitwhere="lmr"value="word"/>will achieve the same result.Transformation is applied in the following order:Replacing charactersSplitting tokensReplacing tokensWhen splitting tokens, longer ones shadow shorter ones. Token replacement
instructions may contradict each other locally, but in entire set they must
converge so that each token has only one replacement option (otherwise
ValueError exception will be thrown).UsageimportsicFor detailed description of all function and methods, see comments in the
source code.Classsic.ModelThis class is designed to instanly create tokenization rules directly in
Python. It is neither convenient nor recommended for complex normalization
tasks, but can be handy for small ones where using external XML config might
seem an overkill.# instantiate Modelmodel=sic.Model()# model is case-sensitivemodel.case_sensitive=True# model will do nothingmodel.bypass=TrueMethodsic.Model.add_ruleadds single tokenization instruction to the
Model instance:# equivalent to XML <split where="lmr" value="beta" />model.add_rule(sic.SplitToken('beta','lmr'))# equivalent to XML <token to="good" from="bad" />model.add_rule(sic.ReplaceToken('bad','good'))# equivalent to XML <character to="z" from="a" />model.add_rule(sic.ReplaceCharacter('a','z'))NB: in case newsic.ReplaceTokenorsic.ReplaceCharinstruction
contradicts something that is already in the model, the newer instruction
overrides older instruction:model.add_rule(sic.ReplaceToken('bad','good'))model.add_rule(sic.ReplaceToken('bad','better'))"bad" --> "good" will not be used; "bad" --> "better" will be used insteadMethodsic.Model.remove_ruleremoves single tokenization instruction from
Model instance if it is there:model.remove_rule(sic.ReplaceToken('bad','good'))# tokenization rule that fits definition above will be removed from modelClasssic.BuilderFunctionsic.Builder.build_normalizer()reads tokenization config,
instantiatessic.Normalizerclass that would perform tokenization according
to rules specified in a given config, and returns thissic.Normalizerclass
instance.ARGUMENTTYPEDEFAULTDESCRIPTIONendpointstr, ModelNonePath to tokenizer configuration file.# create Builder objectbuilder=sic.Builder()# create Normalizer object with default set of rulesmachine=builder.build_normalizer()# create Normalizer object with custom set of rulesmachine=builder.build_normalizer('/path/to/config.xml')# create Normalizer object using ad hoc modelmodel=sic.Model()model.add_rule(sic.SplitToken('beta','lmr'))machine=builder.build_normalizer(model)Classsic.NormalizerMethodsic.Normalizer.save()saves data structure from instance ofsic.Normalizerclass to a specified file (pickle).ARGUMENTTYPEDEFAULTDESCRIPTIONfilenamestrn/aPath and name of file to write.Functionsic.Normalizer.load()reads specified file (pickle) and places
data structure insic.Normalizerinstance.ARGUMENTTYPEDEFAULTDESCRIPTIONfilenamestrn/aPath and name of file to read.Functionsic.Normalizer.normalize()performs string normalization
according to the rules ingested at the time of class initialization, and
returns normalized string.ARGUMENTTYPEDEFAULTDESCRIPTIONsource_stringstrn/aString to normalize.word_separatorstr' 'Word delimiter (single character).normalizer_optionint0Mode of post-processing.control_characterstr'\x00'Character masking word delimiter (single character)word_separator: Specified character will be considered a boundary between
tokens. The default value is' '(space) which seems reasonable choice for
natural language. However any character can be specified, which might be more
useful in certain context.normalizer_option: The value can be either one of0,1,2, or3and
controls the way tokenized string is post-processed:VALUEMODE0No post-processing.1Rearrange tokens in alphabetical order.2Rearrange tokens in alphabetical order and remove duplicates.3Remove all added word separators.control_character: Implementation detail - character that used as word
delimiter inserted in a parsed string at the run time. If parsed string
initially included this character somewhere, normalization will return error.
The value is set to\x00by default.Propertysic.Normalizer.resultretains the result of last call forsic.Normalizer.normalizefunction as dict object with the following keys:KEYVALUE TYPEDESCRIPTION'original'strOriginal string value that was processed.'normalized'strReturned normalized string value.'map'list(int)Map between original and normalized strings.'r_map'list(list(int))Reverse map between original and normalized strings.sic.Normalizer.result['map']: Not onlysic.Normalizer.normalize()generates
normalized string out of originally provided, it also tries to map character
indexes in normalized string back on those in the original one. This map is
represented as list of integers where item index is character position in
normalized string and item value is character position in original string. This
is only valid whennormalizer_optionargument forsic.Normalizer.normalize()call has been set to 0.sic.Normalizer.result['r_map']: Reverse map between character locations in
original string and its normalized reflection (item index is character position
in original string; item value is list [x,y] wherexandyare
respectively lowest and highest indexes of mapped character in normalized
string).Methodsic.build_normalizer()sic.build_normalizer()implicitly creates single instance ofsic.Normalizerclass accessible globally fromsicnamespace. Arguments are same as forsic.Builder.build_normalizer()function.Methodsic.save()sic.save()saves data structure stored in global instance ofsic.Normalizerclass to a specified file (pickle). Arguments are same as forsic.Normalizer.save()method.Functionsic.load()sic.load()reads specified file (pickle) and places data structure in global
instance ofsic.Normalizerclass stored in that file. Arguments are same as
forsic.Normalizer.load()function.Functionsic.normalize()sic.normalize(*args, **kwargs)either uses global classsic.Normalizeror
instantly creates new localsic.Normalizerclass, and uses it to perform
requested string normalization.ARGUMENTTYPEDEFAULTDESCRIPTIONsource_stringstrn/aString to normalize.word_separatorstr' 'Word delimiter (single character).normalizer_optionint0Mode of post-processing.control_characterstr'\x00'Character masking word delimiter (single character)tokenizer_configstrNonePath to tokenizer configuration file.Iftokenizer_configargument is not provided, the function will use global
instance ofsic.Normalizerclass (will create it if it is not initialized).Methodsic.reset()sic.reset()resets globalsic.Normalizerinstance toNone, forcing
subsequently calledsic.normalize()to create new global instance again if it
needs it.Attributesic.result, functionsic.result()sic.resultattribute retains the value ofsic.Normalizer.resultproperty
that belonged to most recently usedsic.Normalizerinstance accessed fromsic.normalize()function (either global or local).Python 3.6 does not supportPEP-562(module attributes). So in Python 3.6, use functionsic.result()rather than
attributesic.result:sic.result()# will work in Python >= 3.6sic.result# will work in Python >= 3.7ExamplesBasic usageimportsic# create Builder objectbuilder=sic.Builder()# create Normalizer object with default set of rulesmachine=builder.build_normalizer()# using default word_separator and normalizer_optionx=machine.normalize('alpha-2-macroglobulin-p')print(x)# 'alpha - 2 - macroglobulin - p'print(machine.result)"""{'original': 'alpha-2-macroglobulin-p','normalized': 'alpha - 2 - macroglobulin - p','map': [0, 1, 2, 3, 4, 5, 5, 6, 6, 7, 7, 8, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 21, 22, 22],'r_map: [[0, 0], [1, 1], [2, 2], [3, 3], [4, 4], [5, 6], [7, 8], [9, 10], [11, 12], [13, 13], [14, 14], [15, 15], [16, 16], [17, 17], [18, 18], [19, 19], [20, 20], [21, 21], [22, 22], [23, 23], [24, 24], [25, 26], [27, 28]]}"""Custom word separatorx=machine.normalize('alpha-2-macroglobulin-p',word_separator='|')print(x)# 'alpha|-|2|-|macroglobulin|-|p'Post-processing options# using normalizer_option=1x=machine.normalize('alpha-2-macroglobulin-p',normalizer_option=1)print(x)# '- - - 2 alpha macroglobulin p'# using normalizer_option=2x=machine.normalize('alpha-2-macroglobulin-p',normalizer_option=2)print(x)# '- 2 alpha macroglobulin p'# using normalizer_option=3# assuming normalization config includes the following:# <setting name="cs" value="0" /># <split value="mis" where="l" /># <token to="spelling" from="speling" />x=machine.normalize('Misspeling')print(x)# 'Misspelling'Using implicitly instantiated classes# normalize() with default instancex=sic.normalize('alpha-2-macroglobulin-p',word_separator='|')print(x)# 'alpha|-|2|-|macroglobulin|-|p'# custom configuration for implicitly instantiated normalizersic.build_normalizer('/path/to/config.xml')x=sic.normalize('some string')print(x)# will be normalized according to config at /path/to/config.xml# custom config and normalization in one linex=sic.normalize('some string',tokenizer_config='/path/to/another/config.xml')print(x)# will be normalized according to config at /path/to/another/config.xmlSaving and loading compiled normalizer to/from diskmachine.save('/path/to/file')# will write /path/to/fileanother_machine=sic.Normalizer()another_machine.load('/path/to/file')# will read /path/to/fileAdding normalization rules to already compiled model# (assuming `machine` is sic.Normalizer instance armed with tokenization ruleset)new_ruleset=[sic.ReplaceToken('from','to'),sic.SplitToken('token','r')]new_ruleset_string=''.join([rule.decode()forruleinnew_ruleset])machine.make_tokenizer(new_ruleset_string,update=True)# rules from `new_ruleset` will be added to the normalizer