mitsuhiko/unicode-literals-pep — Issue #4: tokenizing hook fails with AssertionError on a Django source file
Author: vsajip
Created: 2012-03-05 09:05:15+00:00
State: open

I'm not sure if I've done something wrong, but I get an AssertionError when trying to run the tokenizer on a Python file from the Django source. The gist at https://gist.github.com/1977558 has the files concerned:

* test_tokenize.py - the script which fails
* tokenize_example.py - the file being tokenized and untokenized. This is from the Django source: django/extras/csrf_migration_helper.py
* tokenizer.py - your tokenize module, I renamed it because I was working in /tmp and didn't want to import the Python 3.2 stdlib's tokenize.py
* The test output shows that the tokenize_example module imports OK in Python 2.7.2, but running the test_tokenize script on it with Python3.2 fails with an AssertionError.