Package: pyllk
PyllkA python LL(k) parser with a twist where tokens can be any arbitrary objects. The current implementation uses the backtracking algorithm. I'll eventually implement predictive parsing but the API is designed to remain backward compatible.Important: It is not a tokenizer. It comes with a very basic string tokenizer that assumes a single character is a token. The whole point is to be able to use a third party tokenizer so that you can match them to terminal tokens. UseTokenStreamto push the tokenized list into the parser.Installationpip install pyllkExample. A simple calculatorThis is an example of the typical calculator. In this case, we use characters as tokens. The idea is to parse things like((10+2)*15)/(7-2)and be able to pull the result.Here are the steps involved in the example:Create the function that converts terminal token definitions into instances of instances of pyllk.TerminalTokenCreate the fnuction that converts action definitions into functions to executeCreate the EBNF grammar for the calculatorExecuteConvert terminal token definitionsIn this simple text-based example, terminal tokens are simply a single character. Thematchesfunction
simply needs to execute the regular expression to determine if a give character matches this terminal. The token
definition is set as JSON in the EBNF grammar. The attributes are completely arbitrary and you just need to use
them here. In this example, we chose to use a single attribute calledregexthat describes the regular expression.importosimportreimportpyllkfrompyllk.parserimportContext,Parserfrompyllk.tokenimportTerminalTokenclassCharacterTerminalToken(TerminalToken):def__init__(self,config):self.regex=f"^{config['regex']}$"super(CharacterTerminalToken,self).__init__(self.regex)defmatches(self,obj):returnTrueifre.search(self.regex,f"{obj}")elseFalsedefcharacter_tokenizer(definition:dict)->CharacterTerminalToken:returnCharacterTerminalToken(definition)Convert action definition (i.e. dict) into appropriate functionsIn this case, we use 5 functions. One that pushes numbers into the stack and four for the various operators we want to support.
The action definition will be specified as JSON in the EBNF grammar. It's completely arbitrary, in this case, we chose to use
thefnattribute to indicate which function we're trying to execute. It then returns the corresponding function as an action to
execute when the production rule matches. In order to do this, it uses the context object (an instance of Context, aka a dict) to
carry the state of the FSM.defmake_action(definition:dict)->None:defpush_number(ec):s=""fortokeninec.tokens:s=s+token.representationec.context['stack'].append(float(s))defadd(e):b=e.context['stack'].pop()a=e.context['stack'].pop()e.parser.log("Ex:{}+{}".format(a,b))e.context['stack'].append(a+b)defsub(e):b=e.context['stack'].pop()a=e.context['stack'].pop()e.parser.log("Ex:{}-{}".format(a,b))e.context['stack'].append(a-b)defmul(e):b=e.context['stack'].pop()a=e.context['stack'].pop()e.parser.log("Ex:{}x{}".format(a,b))e.context['stack'].append(a*b)defdiv(e):b=e.context['stack'].pop()a=e.context['stack'].pop()e.parser.log("Ex:{}/{}".format(a,b))e.context['stack'].append(a/b)ifdefinition["fn"]=='push_number':returnpush_numberelifdefinition["fn"]=='add':returnaddelifdefinition["fn"]=='sub':returnsubelifdefinition["fn"]=='mul':returnmulelifdefinition["fn"]=='div':returndivelse:returnNoneThe grammar itselfThis defines a bunch of terminal tokens. As stated above, it uses JSON with a single attribute calledregex. Similarly, actions are
describes as JSON with a single attribute calledfn. See comments in the above sections. Actions are executed when the production rule they annotate resolves.DIGIT = { regex: '\d' }
LPAREN = { regex: '\(' }
RPAREN = { regex: '\)' }

PLUS = { regex: '\+' }
MINUS = { regex: '\-' }
MULT = { regex: '\*' }
DIV = { regex: '\/' }

parser : calc

number : DIGIT number
       | DIGIT

expr : LPAREN calc RPAREN
     | number                      {'fn': 'push_number'}

calc : expr PLUS expr              {'fn': 'add'}
     | expr MINUS expr             {'fn': 'sub'}
     | expr MULT expr              {'fn': 'mul'}
     | expr DIV expr               {'fn': 'div'}Executegrammar=pyllk.load(path_to_grammar_file,token_fn=character_tokenizer,action_fn=make_action)context=Context()context.stack=[]parser=Parser(grammar)parser.parse_string("((10+2)*15)/(7-2)",context)result=context.stack.pop()print(result)Dev setupRunmake init