alex/rply — pull request #18 by DasIch (2014-01-18 14:45:35+00:00)
Title: [WIP] StackedLexer

I've found that the basic LexerGenerator provided by rply while convenient is too simple for some uses cases. So I've implemented this somewhat more powerful StackedLexer, inspired by pygments.

The basic idea is that instead of having rules directly associated with a lexer, the lexer instead has a set of states and each state has a set of rules. Further rules are extended with information about transitions.

During lexing the lexer maintains a stack of states and uses the rules of the state at the top of the stack. Once a rule matches, the lexer performs a transition, if so defined by that rule.

Possible transititions are pushing a new state to the stack or popping the current state of the stack. While I haven't implemented that, a switch to another state without modifying the stack would also be trivial to implement.

This can be very useful for lexing nested comments (see the test for an example) or strings as a sequence of character and escape sequence tokens.

I've made this pull request not as a request to pull the code as-is but rather to get feedback on the idea, API and implementation. Is there interest in this idea? Is the API as I've designed it acceptable or should it be changed? The StackedLexer represents a superset of Lexer and could also replace it entirely without breaking backwards compatibility, what are your thoughts on that?

At the moment quite a bit of code is duplicated between the two lexers, obviously this is not acceptable and needs improvement but I wanted to get feedback before putting too much time into this.