Package: livelex
This module parses text into tokens, and is able to reparse only modified parts
of the text, using the earlier generated tokens. Tokenized text lives in a tree
structure with powerful quering methods for finding tokens and contexts.The livelex module is designed to be fast, and can tokenize in a background
thread, so that even when using very large documents, GUI applications that
need to be responsive do not grind to a halt.Main use case: syntax highlighting in text editors, but also understanding the
meaning of text to be able to provided context sensitive editing features.The livelex module is written and maintained by Wilbert Berendsen.Homepage:https://github.com/wbsoft/livelexDownload:https://pypi.org/project/livelex/The module is designed to parse text using rules, which are regular-expression
based. Rules are grouped into lexicons, and lexicons are grouped into a
Language object. Every lexicon has its own set of rules that describe the text
that is expected in that context.A rule consists of three parts: a pattern, an action and a target.The pattern is a either a regular expression string, or an object that
inherits Pattern. In that case its build() method is called to get the
pattern. If the pattern matches, a match object is created. If not,
the next rule is tried.The action can be any object, and is streamed together with the matched part
of the text. It can be seen as a token. If the action is an instance of
DynamicAction, its filter_actions() method is called, which can yield zero or
more tokens.  The specialskipaction skips the matching text.The target is a list of objects, which can be integer numbers or references
to a different lexicon. A positive number pushes the same lexicon on the
stack, while a negative number pops the current lexicon(s) off the stack, so
that lexing the text continues with a previous lexicon. It is also possible
to pop a lexicon and push a different one.Instead of a list of objects, a DynamicTarget object can also be used, which
can change the target based on the match object.Using a special rule, a lexicon may specify a default action, which is
streamed with text that is not recognized by any other rule in the lexicon.
A lexicon may also specify a default target, which is chosen when no rule
matches the current text.ParsingParsing (better: lexing) text always starts in a lexicon, which is called the
root lexicon. The rules in that lexicon are tried one by one. As soon as there
is a match, a Token is generated with the matching text, the position of the
text and the action that was specified in the rule. And if a target was
specified, parsing continues in a different lexicon.The tokens are put in a tree structure. Every active lexicon creates a Context
list that holds the tokens and child contexts. If a target pops back to a
previous lexicon, the previous context becomes the current one again.All tokens and contexts point to their parents, so it is possible to manipulate
and query the tree structure in various ways.The structure of the tree is built by the TreeBuilder, see thetreeand thetreebuildermodule. At the root is the Context carrying the root lexicon.
The root context contains Tokens and/or other Contexts.The TreeBuilder is capable of tokenizing the text in a background thread and
also to rebuild just a changed part of the text, smartly reusing earlier
generated tokens if possible.Iterating and QueryingBoth Token and Context have many methods for iterating over the tree, for
getting at the parent, child or sibling nodes. Context has various find()
methods to quickly find a token or context at a certain position in the text.Using the Context.query property you can build XPath-like chains of filtering
queries to quickly find tokens or contexts based on text, action or lexicon.
This is described in thequerymodule.ExampleHere are some examples of how to create a Language class and then use it:importlivelexfromliveleximport(Language,lexicon,words,bygroup,bymatch,bytext,default_action,default_target,skip,tomatch,)classMyLang(Language):"""A Language represents a set of Lexicons comprising a specific language.

    A Language is never instantiated. The class itself serves as a namespace
    and can be inherited from.



    """@lexicon(re_flags=0)defroot(cls):yieldr'"',"string",cls.stringyieldr'\(',"paren",cls.parenthesizedyieldr'\d+',"number"yieldr'%',"comment",cls.commentyieldr'[,.!?]',"punctuation"yieldr'\w+',"word"@lexicondefstring(cls):yieldr'\\[\\"]','string escape'yieldr'"',"string",-1yielddefault_action,"string"@lexicon(re_flags=re.MULTILINE)defcomment(cls):yieldr'$',"comment",-1yieldr'XXX|TODO',"todo"yielddefault_action,"comment"@lexicondefparenthesized(cls):yieldr'\)',"paren",-1yield fromcls.root()s=r"""
This is (an example) text with 12 numbers
and "a string with \" escaped characters",
and a% comment that TODO lasts until the end
of the line.
""">>>importlivelex>>>tree=livelex.root(MyLang.root,s)>>>tree.dump()<ContextMyLang.rootat1-144(20children)>├╴<Token'This'at1(word)>├╴<Token'is'at6(word)>├╴<Token'('at9(paren)>├╴<ContextMyLang.parenthesizedat10-21(3children)>│├╴<Token'an'at10(word)>│├╴<Token'example'at13(word)>│╰╴<Token')'at20(paren)>├╴<Token'text'at22(word)>├╴<Token'with'at27(word)>├╴<Token'12'at32(number)>├╴<Token'numbers'at35(word)>├╴<Token'and'at43(word)>├╴<Token'"'at47(string)>├╴<ContextMyLang.stringat48-84(4children)>│├╴<Token'a string with 'at48(string)>│├╴<Token'\\"'at62(stringescape)>│├╴<Token' escaped characters'at64(string)>│╰╴<Token'"'at83(string)>├╴<Token','at84(punctuation)>├╴<Token'and'at86(word)>├╴<Token'a'at90(word)>├╴<Token'%'at92(comment)>├╴<ContextMyLang.commentat93-131(3children)>│├╴<Token' comment that 'at93(comment)>│├╴<Token'TODO'at107(todo)>│╰╴<Token' lasts until the end'at111(comment)>├╴<Token'of'at132(word)>├╴<Token'the'at135(word)>├╴<Token'line'at139(word)>╰╴<Token'.'at143(punctuation)>>>>tree.find_token(50)<Token'a string with 'at48(string)>>>>tree.find_token(50).parent<ContextMyLang.stringat48-84(4children)>>>>d=livelex.Document(MyLang.root,s)>>>d<Document'\nThis is (an example) text w...'>>>>print(d.text())Thisis(anexample)textwith12numbersand"a string with\"escaped characters",anda%commentthatTODOlastsuntiltheendoftheline.>>>d[50:56]'string'>>>withd:...d[9:12]='(a "much longer'...d[20:20]='"'...>>>print(d.text())Thisis(a"much longer example")textwith12numbersand"a string with\"escaped characters",anda%commentthatTODOlastsuntiltheendoftheline.>>>d.get_root()[3].dump()<ContextMyLang.parenthesizedat10-34(4children)>├╴<Token'a'at10(word)>├╴<Token'"'at12(string)>├╴<ContextMyLang.stringat13-33(2children)>│├╴<Token'much longer example'at13(string)>│╰╴<Token'"'at32(string)>╰╴<Token')'at33(paren)>