Package: shreducer
Simple compilers in Python for fun and profit.Example Grammars Included in This ProjectIf you want to reuse the examples code, feel free to copy the code into your projects, but
do not introduce dependencies on examples as the examples may be changed or removed without notice.
Build your own grammar and related classes by extending classes provided in theshreducerpackage only.Python 3.6+ type hints string representationFilter expressions for web APIs. Allows parsing of expressions like this:https://your.api?filter=(statuseq open and type eq store) or (status eq closed and type not in office, garage)With a compiler of Elasticsearch queries included (not tested since 2016)A couple of primitive grammars for basic arithmetic expressions as examplesComponentsTokenizers(no need to implement your own, we rely onshlexfrom Python Standard Library)GrammarsParsers(no need to implement your own, we use ashift-reduceparser, hence the name of the project)GeneratorsTokenizersplits input string into lexical units of the grammar (we call them tokens here).Grammardescribes the syntax rules of a language that we want to parse.Parserparses sequence of tokens and generates a parse tree.
For example('+',2,('-',10, 3))could be a parse tree.Generatortakes a parse tree and evaluates it. For example, an arithmetic generator could take
a parse tree('+',2,('-',10, 3))as an input and produce9as an output.All these components together make a compiler.Expressing a New GrammarSee examples undershreducer_examples/:DictG- simplest of all grammars, the most suitable to understand the basic idea, parser produces parsed dictionaryListG- another simple grammar, but unlike dictionary grammar, parser for this one produces parse treePlusMinusArithmeticsG- simple arithmetic expression parser, parser produces parse treeBetterArithmeticsG- arithmetic expression parser that respects operator precedence, parser produces parse treeFilterExpressionsG- comparison operators and logical operators, parser produces parse treeBetterFiltersG- comparatively rich filter expression language, unlike other grammars this one uses look-ahead,
parser produces parse treeTypeHintsG- parsing Python 3.6+ type hints string representationThere is some magic (a meta class) going on intclass to allow
declaring a string constant without writing its value twice:classMyGrammar(Grammar):classt:IDENT=NonePLUS_MINUS='+-'EXPR=()After the classMyGrammaris created, the value ofMyGrammar.t.IDENTwill be"IDENT".
Similarly,MyGrammar.t.PLUS_MINUSwill be"PLUS_MINUS", andMyGrammar.t.EXPRwill be"EXPR".Member ofclass twith valueNoneis treated as the default token type.Members ofclass twith value()are treated as names of tokens of higher order – expressions.Testing Your New GrammarIf you implement just a grammar, you can try parsing input strings withGrammar.simple_parse(which is a class
method).For example, to tryTypeHintsG(grammar for parsing type hints string representation in Python 3.6+),
you can do:print(TypeHintsG.simple_parse('typing.Union[typing.List[str], typing.Dict[str, int]]'))This will produce the following parse tree:{"name":"typing.Union","args":[{"name":"typing.List","args":[{"name":"str","args":None},],},{"name":"typing.Dict","args":[{"name":"str","args":None},{"name":"int","args":None},],},],}