Package: parsergen
parsergenA simple library for creating parsers and lexers.Quickstartpip install parsergenDefining a LexerTokens have different regular expressions. They can also have modifier functions, for example theINTtokens get their values turned into anint.fromparsergenimport*classCalcLexer(Lexer):@token(r"0x[0-9a-fA-F]+",r"[0-9]+")defINT(self,t):ift.value.startswith("0x"):t.value=int(t.value[2:],base=16)else:t.value=int(t.value)returntADD=r"\+"SUB=r"\-"POW=r"\*\*"# must be first, as is longer than 'MUL' token!MUL=r"\*"DIV=r"\/"SET=r"set"TO=r"to"ID=r"[A-Za-z_]+"LPAREN=r"\("RPAREN=r"\)"ignore="\t"ignore_comment=r"\#.*"Creating a ParserGrammar ExpressionsGrammar Expressions describe the syntax that can be parsed.
For our basic example calculator, you will get a terminal to type math expressions> 2 + 3 * 4
14
> (2 + 3) * 4
20
> 2 ** 2 ** 3
256It is important that the precedence of the arithmetic operators is correct, we have to account for this when designing our grammar rules.
Here is the grammar:statement       :  assign | expr
assign          :  SET ID TO expr
expr            :  prec3
prec3           :  prec2 (ADD | SUB prec2)*
prec2           :  prec1 (MUL | DIV prec1)*
prec1           :  factor (POW prec1)?
factor          :  INT | ID
factor          :  LPAREN expr RPARENthe rulesprec3andprec2are left associative, whereasprec1is right associative because it implements the pow operator
We can then define our parser.classCalcParser(Parser):tokens=CalcLexer.tokensstarting_point="statement"def__init__(self):self.names={}@grammar("assign | expr")defstatement(self,p):print(p[0])@grammar("SET ID TO expr")defassign(self,p):self.names[p[1]]=p[3]@grammar("prec3")defexpr(self,p):returnp[0]@grammar("prec2 (ADD | SUB prec2)*")# left associativedefprec3(self,p):r=p[0]forop,numinp[1]:ifop=="+":r+=numelse:r-=numreturnr@grammar("prec1 (MUL | DIV prec1)*")# left associativedefprec2(self,p):r=p[0]forop,numinp[1]:ifop=="*":r*=numelse:r/=numreturnr@grammar("factor (POW prec1)?")# right associativedefprec1(self,p):ifp[1]:returnp[0]**p[1][1]returnp[0]@grammar("INT")deffactor(self,p):returnp[0]@grammar("ID")deffactor(self,p):try:returnself.names[p[0]]exceptKeyError:raiseException(f"variable '{p[0]}' is not defined.")@grammar("LPAREN expr RPAREN")deffactor(self,p):returnp[1]# We can then create a simple runtime loopl=CalcLexer()p=CalcParser()whileTrue:s=input("> ")l_result=l.lex_string(s)try:p.parse(l.lex_string(s))exceptExceptionase:print(e)Handling NewlinesThe Lexer, by default knows nothing about line numbers. You have to tell it what to do.classMyLexer(Lexer):@token(r"\n+")defNEWLINE(self,t):self.lineno+=len(t.value)self.column=0returnt...Overcoming issues with left recursionRecursive descent parsers are unable to handle direct or indirect left recursion. This is an issue when writing expressions for left associative operators.
The following example is directly left recursive:expr  :  expr PLUS termand when attempting to process this rule it will fall into an infinite loop.
There are different ways to solve this problem, my solution is below:expr  :  term (PLUS term)*The disadvantage to this is that there is then some processing required after the pattern matching to reach the original desired strucutre or action.@grammar("term (PLUS term)*")defexpr(self,p):rv=p[0]forop,terminp[1]:rv+=termreturnrvSeeherefor more details.Writing expressions for right-associative operatorsSome operators are right associative, for example the**operator.
Right recursion can be implemented more normally in the grammar expression:expr  :  term (POW expr)?This behaves as expected, after pattern matching you do have to perform a check in your code as seen next:@grammar("term (POW expr)?")defexpr(self,p):ifp[1]:returnp[0]**p[1][1]returnp[0]Printing the grammar for your parserIt is sometimes helpful to see the entire grammar for you parser. This can be done as shown below:fromparsergenimportget_grammarprint(get_grammar(CalcParser))Seeexample_calc.pyandexample.pyfor more examples, or look at the source code.