Engelberg/instaparse — pull request #45 by mnemnion (2013-08-16 15:35:09+00:00)
Title: insta/re-parse

Hi Mark,

I needed this function for a project I'm working on, and I thought it was broadly useful enough to propose for the API. I suspect this function will get repeatedly rewritten if not provided.

Two use cases, and I'm using both already. The first is macro expansion: there might be macros in the macros, so we have to do insta/transform as many times as necessary until there aren't any macros left. Then, we need to completely flatten everything and reparse with the same rules.

The second is embedded code: Imagine a simple config language, where one option may contain some Scheme. It may not even be in there, and we might have a lot of config to do quickly, say on a boot. The config language is designed to be parsed with regular expressions, which is common. The parser will run faster if we just scoop the Scheme and parse it in a separate pass, with a different grammar.

An elegant extension of the grammar is possible with this function, namely progressive refinement. Currently if you provide more than one rule with the same name, Instaparse uses the last one it encounters. Actually you may want to retain this behavior, since it lets us hot-load a different version of a rule. The progressive refinement option is to parse the tree using the first set of definitions of all rules. Then, if there are remaining rules, it `transform`s the collected nodes against the next set of definitions, and so on if there are more than two. I *think* this doesn't require a separate pass, conceptually, but insta/re-parse makes it cheap to build this way. There are tradeoffs.