Adds test for tokenizer dealing with missing spaces and numbers in variable names.
Fixes many bugs in the tokenizer and deals (and adds test) for the block comments right after each other.
Uses tokenizier.hasMoreText() instead of hasMoreTokens sinces that's what we're really checking.
Fixes integerConstant finding code to not consume adjacent text.
Adds exception when trying to create integerConstants over 32767.
Cleans up.
Adds test for underscores in identifiers.
Cleans up tokenizer and tokenizer tests. Allows for newlines is strings.
Massive refactoring of tokenizer and tests.
Massive refactoring of Tokenizer. Adds line number to tokens. Using correct name 'tag' instead of 'type'.
DRYs up tokenizer tests a bit.
Beginning of building an AST.