Added support for more tokens.
Fixed some string and comment corner cases.