Changed lexer to support iterator interface

Motivation: allows us to iterate over tokens like items in any other collection