Add preceding whitespace collection to tokenizer
Merge @joel-wright's whitespace tokenizer branch. Thanks!
Fix a few tokenize tests and merge them back together.
Rename test_tokenizer to test_tokenize.
Small tokenizer changes & tokens now have a prefix attribute instead of preceeding_whitespace.
Allow both the old tokenizer and the new one (able to toggle).
The new tokenizer is more or less working now. Indents are calculated as they should
Trying to move the indent/dedent logic back into the tokenizer.
Check parentheses level in tokenizer.
classes and functions are new statements and should never get removed by the error recovery.
Working with dedents in error recovery.
invalid.py test changes. Error recovery will be different from the old one.
The func/class dictionaries must be changed if some scopes are removed by the parser's error recovery.
Fix invalid test issues.
Fixes for issues with empty compound_stmt. We always remove a whole stmt and just a funcdef as an error correction.
suites without indent can also be deleted.
dynamic_params correction.
Don't use NotFoundError anymore, since it's very ambiguous what that would imply.
Fix positioning of the user statements.
Fixed list comprehension name lookups.