Package: pymetaterp
pymetaterpThis is a python AST builder that uses no Python modules. A longer stackless version is available for easier porting.single_file.pyis a stand-alone 502 lines script.Its (also) just another parsing expression grammar (PEG) based parser with one major difference. The parsed grammar is interpreted instead of compiled. This makes it easy to modify the language (by editing its grammar)as well asthe language that grammar is written in (and the language ofthatgrammar).[This is apre-releaseof sorts. There are probably some errors and missing information.]Download and rungit clone https://github.com/asrp/pymetaterp
cd pymetaterp
python single_file.pyorpython single_file.py filename.pyThis will print out the AST of the given file (orsingle_file.py's own AST). Sample beginning of the output:file_input
  regular_assign
    testlist
      NAME
        str 'NAME'
      NAME
        str 'FLAGS'To run files from the librarypython test/boot_test.py 
python test/python_parse_test.pyFilessingle_file.pyis mainly for demonstration. This module is otherwise separated into files. There are many files but they are mostly separate. The import dependencies isutil.py
  boot.py
  boot_stackless.py
    python.pyOther files have no imports. To get something useful, you'll have to import multiple files. Seetest/python_parse_test.pyandtest/boot_test.pyfor some examples.ReplAn obvious thingmissingis the grammar read-eval-print loop (repl) so the interpreter can be fed one rule at a time, parsing subsequence input using the rules seen so far.Source reading orderI'd suggest readingboot.pyandbootstrapinboot_grammar.pyfirst. The two form the core and together withboot_tree.py, they can regenerateboot_tree.Thenboot_stacklessis the same asboot.pybut doesn't use the Python call stack/recursion for parsing.python.pyadds functionality to theboot.pyinterpreter.diffinboot_grammar.pyadds the syntax for those.Finally,python_grammar.pycontains the python grammar to be finally parsed.Python language parsedThe module builds the AST for Python 2.x programs. It is able to parse all of Python 2.x (in fact, it contains a slightly modified version of the Python 2.x grammar) but is less lenient with whitespaces. For example, parsingfrom my_module import (var1, var2,
                       var3, var4)gives an error.Since this is a pre-release, there are likely bugs with parts of the language I don't use so often. Itcanbuild the AST for all files included here.Gramamr language differencesThe beginning ofboot_grammar.pyself-describes the grammar. Its a PEG so all "or" (|) returns the first match and "and" and "quantified" (*, +, ?) are greedy.name = (letter | '_') (letter | digit | '_')*
expr = apply | exactly | token | parenthesis | output

exactly! = "'" {(escaped_char | ~'\'' anything)*} "'"
token! = "\"" {(escaped_char | ~'"' anything)*} "\""
escaped_char! = '\\' {'n'|'r'|'t'|'b'|'f'|'"'|'\''|'\\'}
apply! = ('\t'|' ')* {name}
parenthesis = "(" {or} ")"
output! = "{" {or} "}"

not = "~" {expr=negation} | expr
quantified = not (('*' | '+' | '?')=quantifier)?
bound = quantified ('=' {name=inline})?
and = bound*
or = and ("|" {and})*

rule = spaces {name=rule_name '!'?=flags and=args ("=" {or})}
grammar = {rule*} spacesThe main difference from other PEG.output rule:a {b c} dwill match the concatenation ofa b c dbut only return what matchedb c.quantifier collapse:letter letter*returns a list rather than a pair with the second element being a list matchingletter*.nested and collapse:a (b (c d)) ehas the same output asa b c d e(see inline below if some pairs need to be explicitly grouped).node collapsing: nodes in the output with only one child are replaced by their parent, unless the!("don't collapse") flag is set for that node.inline: shamelessly takenOhmbut with a slightly different interpretation.expression=namecreates a node namednamewrapping the output ofexpression.rule replacement: having a secondrule_name = expressionline replaces the first definition ofrule_name(instead of appending into an or).two basic tokens: there are two basic token types:'a'(single quote) and"a"(double quote). The double quoted token allows whitespace before matching.Regenerating boot_tree.pyCreate some treematch_treeusingInterpreter.matchand callsaveon the result.match_tree.save("tree.py")Left recursionWhile "PEG/packrat parsers can support left-recursion", the tree output isn't the one we want. The python functionsreformat_binaryandreformat_atomfixes a parsed tree's ouput.Source odditiesTwo hard-coded rulesif root[NAME] == "anything":
            return pop(self.input)
        elif root[NAME] == "void":
            returnHard-coded semantics for tokensif name == "token":
            while pop(self.input) in self.whitespace:
                if self.input[0][self.input[1]] == '\\':
                    pop(self.input)
            self.input[1] -= 1
        if name == "token" and root[0].isalpha():
            top = pop(self.input)
            if top.isalnum() or top == '_':
                raise MatchError("Prefix matched but didn't end.")
            self.input[1] -= 1OptimizationSome effort were made to make these files short (especiallysingle_file.py) but not too much. There are still some asserts around and commented print statements that can be useful for debugging. The final goal is, of course, to reduce the program's complexity and verbosity, not its line count.Missing featuresFeatures/bloat from a longer version of this program not (yet?) moved over:Debugging tree of nodes visited and their input and outputFunction arguments (its in the grammar but not the interpreter)Nested list inputs (its also in the grammar but not the interpreter)name, args, flags, body as parameters instead of positional childrenMemoizationMatched input start and end positionsExact python expression matching for predicate, action and rule value.balancedis used as a simpler heuristic for now.Removing featuresTo get a smaller file with just the basics.patch -R pymetaterp/python.py < patches/python_pos.patch
patch -R pymetaterp/python.py < patches/python_memoizer.patch
patch -R pymetaterp/boot_stackless.py < patches/boot_pos.patch
patch -R pymetaterp/boot_stackless.py < patches/boot_memoizer.patchReadingsOmeta- Warth's thesis reads very well.PEG and packrat parserPackrat Parsers Can Support Left RecursionOther similar projectsparsimoniousPymetapyparsing