Yacc is dead

We present two novel approaches to parsing context-free languages. The first approach is based on an extension of Brzozowski’s derivative from regular expressions to context-free grammars. The second approach is based on a generalization of the derivative to parser combinators. The payoff of these techniques is a small (less than 250 lines of code), easy-to-implement parsing library capable of parsing arbitrary context-free grammars into lazy parse forests. Implementations for both Scala and Haskell are provided. Preliminary experiments with S-Expressions parsed millions of tokens per second, which suggests this technique is efficient enough for use in practice. 1 Top-down motivation: End cargo cult parsing “Cargo cult parsing” is a plague upon computing. Cargo cult parsing refers to the use of “magic” regular expressions—often cut and pasted directly from Google search results—to parse languages which ought to be parsed with contextfree grammars. Such parsing has two outcomes. In the first case, the programmer produces a parser that works “most of the time” because the underlying language is fundamentally irregular. In the second case, some domain-specific language ends up with a mulish syntax, because the programmer has squeezed the language through the regular bottleneck. There are two reasons why regular expressions are so abused while context-free languages remain foresaken: (1) regular expression libraries are available in almost every language, while parsing libraries and toolkits are not, and (2) regular expressions are “WYSIWYG”—the language described is the language that gets matched—whereas parser-generators are WYSIWYGIYULR(k)—“what you see is what you get if you understand LR(k).” To end cargo-cult parsing, we need a new approach to parsing that: 1. handles arbitrary context-free grammars; 2. parses efficiently on average; and 3. can be implemented as a library with little effort. The end goals of the three conditions are simplicity, feasibility and ubiquity. The “arbitrary context-free grammar” condition is necessary because programmers will be less inclined to use a tool that forces them to learn or think about LL/LR arcana. It is hard for compiler experts to imagine, but the constraints on LL/LR 1 The term “cargo cult parsing” is due to Larry Wall, 19 June 2008, Google Tech Talk. grammars are (far) beyond the grasp of the average programmer. [Of course, arbitrary context-free grammars bring ambiguity, which means the parser must be prepared to return a parse forest rather than a parse tree.] The “efficient parsing” condition is necessary because programmers will avoid tools branded as inefficient (however justly or unjustly this label has been applied). Specifically, a parser needs to have roughly linear behavior on average. Because ambiguous grammars may yield an exponential number of parse trees, parse trees must be produced lazily, and each parse tree should be paid for only if it is actually produced. The “easily implemented” condition is perhaps most critical. It must be the case that a programmer could construct a general parsing toolkit if their language of choice doesn’t yet have one. If this condition is met, it is reasonable to expect that proper parsing toolkits will eventually be available for every language.When proper parsing toolkits and libraries remain unavailable for a language, cargo cult parsing prevails. This work introduces parsers based on the derivative of context-free languages and upon the derivative of parser combinators. Parsers based on derivatives meet all of the aforementioned requirements: they accept arbitrary grammars, they produce parse forests efficiently (and lazily), and they are easy to implement (less than 250 lines of Scala code for the complete library). Derivative-based parsers also avoid the precompilation overhead of traditional parser generators; this cost is amortized (and memoised) across the parse itself. In addition, derivative-based parsers can be modified mid-parse, which makes it conceivable that a language could to modify its own syntax at compileor run-time. 2 Bottom-up motivation: Generalizing the derivative Brzozowski defined the derivative of regular expressions in 1964 . This technique was lost to the “sands of time” until Owens, Reppy and Turon recently revived it to show that derivative-based lexer-generators were easier to write, more efficient and more flexible than the classical regex-to-NFA-to-DFA generators . (Derivative-based lexers allow, for instance, both complement and intersection.) Given the payoff for regular languages, it is natural to ask whether the derivative, and its benefits, extend to context-free languages, and transitively, to parsing. As it turns out, they do. We will show that context-free languages are closed under the derivative—they critical property needed for parsing. We will then show that context-free parser combinators are also closed under a generalization of the derivative. The net impact is that we will be able to write a derivative-based parser combinator library in under 250 lines of Scala, capable of producing a lazy parse forest for any context-free grammar. 2 The second author on this paper, an undergraduate student, completed the implementation for Haskell in less than a week.