alliedmodders/sourcemod — pull request #45 by dvander (2014-06-22 06:43:44+00:00)
Title: Introduce 4-token lookahead buffer.

The grammar for the Transitional Syntax requires four tokens of lookahead. For example, when encountering the `public` keyword for a method definition, we need to look for ``, `(`, `)`, and `=` to determine whether it's bind-to-existing-native syntax. If it's not, we have to rewind back to the `(` to look for a constructor, or even all the way back to the symbol to check for a type expression.

This patch refactors the lexer to keep two buffers: one for preprocessor lexing, and one for normal lexing. Each buffer can remember four tokens.