Refactor to allow different lexer classes, one for bytes and one for unicode.
Stop relying on length and boolness of token.
Remove unused methods.
Moved decode method from Token to the decoder.
Simplify Token.
Lexing unicode.
Refactor unicode lexer so we can share all code between the two implementations.
Remove unused constructor.
Use ascii codec consistently.
Move setup for lexer test.
Rename some test classes more appropriately.
A few more renames.
Avoid blindly copying test code.
Allow custom decoder in test.
Test decoder also in unicode mode.
Fix lex_it for non-binary mode.
Fix more tests.
Refactor some tests.
Hack around Python bug: UnicodeDecodeError does not like non-binary input, apparently.
Split unicode on non-combining characters for encoder.