Package: compynator
IntroductionCompynator is a tiny (~400 SLOCs), pure Python implementation ofparser
combinators. With this
library, one can build up a complex parser from primitive parsers such as “get
one token” (theOneparser), or compose parsers together such as “if this
parser fails, try that parser” (theOrcombinator), etc. The mental model is
a binary tree of execution nodes through which a sequence of input tokens flows.The implementation in this package supports optional memoization and curtailment
as described in[Frost2007]. This allows a memoized parser to achieve
asymptotically best performance, and support ambiguous grammars.Compynator is not an official Google product.ExamplesAn example that solveshttps://leetcode.com/problems/parsing-a-boolean-expression:t=Terminal('t').value(True)f=Terminal('f').value(False)e=Forward()n=Terminal('!(').then(e).value(lambdax:notx).skip(')')a_empty=Succeed(True)a_e_tail=Forward()a_e_tail.is_(Terminal(',').then(e).then(a_e_tail,lambdax,y:xandy)|a_empty)a=Terminal('&(').then(e).then(a_e_tail,lambdax,y:xandy).skip(')')o_empty=Succeed(False)o_e_tail=Forward()o_e_tail.is_(Terminal(',').then(e).then(o_e_tail,lambdax,y:xory)|o_empty)o=Terminal('|(').then(e).then(o_e_tail,lambdax,y:xory).skip(')')e.is_(t|f|n|a|o)self.assertEqual(e('!(f)'),{Result(True,'')})self.assertEqual(e('|(f,t)'),{Result(True,'')})self.assertEqual(e('&(t,f)'),{Result(False,'')})self.assertEqual(e('|(&(t,f,t),!(t))'),{Result(False,'')})Seetest_benchmark.pyfor a literal translation of
the ABNF rules for URI parsing as given in[RFC3986]into a structured tree:Uri
    Scheme http
    HierPart
        Authority
            UserInfo None
            Host www.ics.uci.edu
            Port None
        Path /pub/ietf/uri/
    Query query
    Fragment fragmentThere are more examples in thetestsdirectory.Translating Augmented Backus-Naur Form (ABNF)Care should be taken to ensure thatrepeattakes all possible results
instead of greedily parsing for the most. For example, the ABNF*3"a" "aa"cannot be simply translated asTerminal('a').repeat(0,3) + 'aa'. This
translation will fail to parseaaaabecause it greedily matches the first
3a’s, then fails to find the remaining 2a’s. The correct translation
should beTerminal('a').repeat(0,3, take_all=True) + 'aa'.ABNFCompynatorTerminalTerminalRuleParserConcatenationp1 + p2Alternativep1 | p2Sequence groupNormal use of parenthesesVariable repetitionp.repeat(lower, upper, take_all)Specific repetitionp.repeat(x, x)Optionalp.repeat(0, 1, take_all)Translating Parsing Expression Grammar (PEG)Unlike ABNF, PEG operators are always greedy. When translating PEG, we do not
need to worry about backtracking the repetitions withtake_all.PEGCompynatorTerminalTerminalNonterminalParserEpsilonEmptySequencep1 + p2Ordered choicep1 | p2Zero or morep.repeat()One or morep.repeat(1)Optionalp.repeat(0, 1)And predicateLookahead(p)Not predicateLookahead(p, take_if=False)Combinator vs GeneratorAdvantagesAdvantages of parser combinators versus parser generators are:Readability. A grammar can be expressed in a very similar form as its BNF.
The code can be considered anexecutable specificationof the grammar.Simple setup. The code is the grammar. There is no need to run a generator to
regenerate code when the grammar changes.Understandability. Each parser is generally short and simple that its
correctness can be easily verified. There is no need to look into generated
code, or the code of the parser generator.Parser combinators support context-sensitive grammars. For example, to parse
an XML body, assumingstartparses a start tag,bodyparses the body,
andendparses a specified end tag:xml_tag=start.then(lambdatag_name:body.skip(end(tag_name)))Combination of lexing and parsing. Most parser generators perform their
lexing and parsing phases separately. Parser combinators combine these phases
together. Hence they are not limited to string inputs. The example (intest_core.py) below takes a tokenized sequence.NUM,OP,TERMINAL=0,1,2tokens=[(NUM,2),(OP,operator.add),(NUM,10),(OP,operator.mul),(NUM,4)]num=One.where(lambdac:c[0]==NUM)op=One.where(lambdac:c[0]==OP).value(lambdac:c[1])mult_div=op.where(lambdac:cin(operator.mul,operator.truediv))add_sub=op.where(lambdac:cin(operator.add,operator.sub))left_paren=One.where(lambdac:c[0]==TERMINALandc[1]=='(')right_paren=One.where(lambdac:c[0]==TERMINALandc[1]==')')expr=Forward()factor=(num.value(lambdat:t[1])|left_paren.then(expr).skip(right_paren))defdo_op(left,op,right):returnop(left,right)term=Forward()term.is_((Collect(term,mult_div,factor).value(lambdav:do_op(*v))^factor).memoize())expr.is_((Collect(expr,add_sub,term).value(lambdav:do_op(*v))^term).memoize())calc=expr.filter(lambdar:notr.remain)self.assertEqual(set(expr(tokens)),{Result(value=42,remain=[]),Result(value=12,remain=tokens[3:]),Result(value=2,remain=tokens[1:]),})self.assertEqual(calc(tokens),Succeed(42)([]))DisadvantagesDisadvantages of parser combinators are:Familiarity. Most textbooks write about parser generators and traditional
parsing techniques such as LL, LR, etc. Parser combinators are more common
in functional and logic programming communities, as popularized by[Wadler1985]and[Hutton1992].Coupling of code and grammar. The downside of simple setup is a tight
coupling of code and grammar, which might make it difficult to understand.As it is implemented here, performance might be impacted due to composition
overhead. Seetest_benchmark.pyfor details. On
the same machine, the result for URI parsing could be ~70 times slower:t.test_parse_uri() 903.5961110000001 usec per run
t.test_urlparse() 13.704007000000074 usec per runAll the advantages and disadvantages of scannerless parsing apply too.LimitationsCurrently, this library does not implement:Source context such as line and column number of the token.“Greedy” matching in the same sense as in regular expression (i.e. longest
match). The greedy operation in this library is in the “greedy algorithm”
sense, i.e. the first rule that matches will be taken.Space treatments. Spaces have to be explicitly taken care of in grammars.References[Wadler1985]Wadler, Philip. (1985). “How to replace failure by a list of
successes”. Proc. conference on functional programming and computer
architecture. Springer–Verlag.[Hutton1992]Hutton, Graham. (1992). “Higher-order functions for parsing”.
Journal of functional programming, 2(3), 323–343.[Frost2007]Frost R.A., Hafiz R., Callaghan P. (2007) “Parser Combinators for
Ambiguous Left-Recursive Grammars”. In: Hudak P., Warren D.S. (eds)
“Practical Aspects of Declarative Languages”. PADL 2008. Lecture Notes in
Computer Science, vol 4902. Springer, Berlin, Heidelberg[RFC3986]Berners-Lee, T., Fielding, R., and L. Masinter, “Uniform Resource
Identifier (URI): Generic Syntax”, STD 66, RFC 3986, January 2005.