Implemented an asynchronous tokenizer that handles whitespace and newlines only
refactored to static ExtractTokens() method that returns a list
Major refactoring to method calls.
added test to remind me where I left off...
Progress with tokenizer on multiline block comment.
fixed one bug anyway...
Corrected buggy test
Got tokenizer working for block comments.
Fixed bug with block comment start inside block comment was tokenized.
Strings are initially working.
clearing out test code from demo program
Fix for calculating the length of escaped quotes in strings.
fixed unknown token parsing
got basic keyword tokenizing working.