Added whitespace detection to the lexer + tests.
Changed token fields to be public.