Package: tokenspan
Tokenization for language processingThis package contains some generic configurable tools allowing to cut a string in sub-parts (cf.Wikipedia), calledToken, and to group them into sequences calledTokens. ATokenis a sub-string from a parent string (say the initial complete text), with associated ranges of non-overlaping characters. The number of associated ranges is arbitrary. ATokensis a collection ofToken. These two classes allow to associate to anyTokena collection of attributes in a versatile way, and to pass these attributes from one object to the next one while cuttingTokeninto sub-parts (collected asTokens) and eventually re-merging them into largerToken.TokenandTokensclasses allow basic tokenization of text, such as word splitting, n-gram splitting, char-gram splitting of arbitrary size. In addition, it allows to associate several non-overlapping sub-strings into a givenToken, and to associate arbitrary attributes to these parts. One can compare two differentTokenobjects in terms of their attributes and/or ranges. One can also apply basic mathematical operations and logic to them (+,-,*,/) corresponding to the union, difference, intersection and symmetric difference implemented by Python set ; here the sets are the ranges of position from the parent string.Depositories, and online documentationThe different sources of informations for this packages are :the official Python Package Installation (PyPI) repository is onhttps://pypi.org/project/tokenspanthe official git repository is onhttps://framagit.org/nlp/tokenspanthe official documentation is onhttps://nlp.frama.io/tokenspan/Philosophy of this libraryIntokenspan, one thinks of a string as a collection of integers: the position of each character in the string. For instance'Simple string for demonstration and for illustration.'# the parent string'01234567891123456789212345678931234567894123456789512'# the positions'       string                       for illustration '# the Span span1'       789112                       678 412345678951 '# the ranges'Simple                                               '# the Span span2'012345                                               '# the rangesTo define theSpan'string for illustration'consists in selecting the positions[range(7,13),range(36,39),range(40,52)]from the parent string, and theSpan'simple'is defined by the positions[range(0,6),].In addition, one can see the above ranges as sets of positins. Then it is quite easy to perform some basic operations on theSpan, for instance the addition of twoSpanstr(span1+span2)='Simple string for illustration'is interpreted as the union of their relative sets of positions.In addition to these logical operations, there are a few utilities, like the possibility to split or slice aSpanintoSpanobjects, as long as their are all related to the same parent string.Basic exampleBelow we give a simple example of usage of theTokenandTokensclasses.importrefromtokenspanimportSpanstring='Simple string for demonstration and for illustration.'initial_span=Span(string)# char-gram generationchargrams=initial_span.slice(0,len(initial_span),3)str(chargrams[2])# return 'mpl'# each char-gram conserves a memory of the initial stringchargrams[2].string# return 'Simple string for demonstration and for illustration.'cuts=[range(r.start(),r.end())forrinre.finditer(r'\w+',string)]spans=initial_span.split(cuts)# this returns a list of Span objects# spans conserve the cutted partsinteresting_spans=spans[1::2]# so one has to take only odd elements# an other possibility to keep only the words is to construct it explicitlyspans=Span(string,ranges=cuts)# n-gram constructionngram=[Span(string,ranges=[r1,r2])forr1,r2inzip(spans.ranges[:-1],spans.ranges[1:])]ngram[2]# return Span('for demonstration', [(14,17),(18,31)])str(ngram[2])# return 'for demonstration'ngram[2].ranges# return [range(14, 17), range(18, 31)]ngram[2].subSpans# return the Span instances composed of span 'for' and span 'demonstration'# are the two 'for' Token the same ?interesting_spans[2]==interesting_spans[-2]# return False, because they are not at the same position# basic operations among Tokenfor_for=interesting_spans[2]+interesting_spans[-2]str(for_for)# return 'for for'for_for.ranges# return [range(14, 17), range(36, 39)]for_for.string# return 'Simple string for demonstration and for illustration.'# to check the positions of the two 'for' Token :#        '01234567890...456...01234567890.....67890............'# also available :# span1 + span2 : union of the sets of span1.ranges and span2.ranges# span1 - span2 : difference of span1.ranges and span2.ranges# span1 * span2 : intersection of span1.ranges and span2.ranges# span1 / span2 : symmetric difference of span1.ranges and span2.rangesOther examples can be found in thedocumentation.Comparison with other Python librariesA comparison with some other NLP librairies (nltk, gensim, spaCy, gateNLP, ...) can be found in thedocumentationInstallationSimply runpipinstalltokenspanshould install the library from Python Package Index (PIP). The official repository is onhttps://framagit.org/nlp/tokenspan. To install the package from the repository, run the following command linesgitclonehttps://framagit.org/nlp/tokenspan.gitcdtokenspan/
pipinstall.Once installed, one can run some tests usingcdtests/
python3-munittest-v(verbosity-vis an option).VersionsSeeCHANGES filein this folder.About usPackage developped for Natural Language Processing at IAM : Unité d'Informatique et d'Archivistique Médicale, Service d'Informatique Médicale, Pôle de Santé Publique, Centre Hospitalo-Universitaire (CHU) de Bordeaux, France.You are kindly encouraged to contact the authors by issue on theofficial repository, and to propose ameliorations and/or suggestions to the authors, via issue or merge requests.Last version : Jan 20, 2022