Simplify incremental lexer implementation: do not use send/yield system.
Iterator fix, raise ValueError if function does not consume all tokens.